[NIXL] ASIO support by podkidyshev · Pull Request #996 · NVIDIA/cloudai

podkidyshev · 2026-08-11T11:09:08Z

Summary

Add ASIO support into NIXL
ETCD isn't run for case when it's not needed (etcd_endpoints is set to an empty string)
- This is not a backwards compatibility violation but rather bug fix. It may result in minor decreases of bench execution time, but other metrics should be the same

Test Plan

Automated CI
Manual runs with the new NIXL scenario

Additional Notes

coderabbitai · 2026-08-11T11:09:48Z

📝 Walkthrough

Walkthrough

Adds ETCD, ASIO, and null-runtime support to NIXLBench. Updates environment and Slurm command generation, adds runtime validation and lifecycle tests, and introduces POSIX, UCX, network, and storage smoke-test configurations with documentation.

Changes

NIXLBench runtime support

Layer / File(s)	Summary
Runtime contracts and capability detection `src/cloudai/workloads/common/nixl.py`, `src/cloudai/workloads/nixl_bench/nixl_bench.py`	Adds managed ETCD endpoint handling, ETCD and ASIO capability properties, validated ASIO settings, runtime-specific command serialization, and conditional environment variables.
Environment and Slurm orchestration `src/cloudai/workloads/nixl_bench/slurm_command_gen_strategy.py`, `tests/workloads/nixl_bench/test_command_gen_strategy_slurm.py`	Generates runtime-specific commands. Adds ASIO process validation, conditional ETCD lifecycle handling, startup delays, image-state cleanup, and coverage for ASIO, managed ETCD, external ETCD, and null-runtime cases.
Benchmark scenarios and documentation `conf/experimental/test/`, `conf/experimental/test_scenario/`, `doc/workloads/nixl_bench.rst`	Adds POSIX and UCX benchmark configurations, network and storage scenarios, and runtime coordination documentation.

Estimated code review effort: 3 (Moderate) | ~30 minutes

Possibly related PRs

NVIDIA/cloudai#991: Updates nixl_bench.py with metrics observation and SOL reporting support.

Suggested reviewers: jj10306

🚥 Pre-merge checks | ✅ 4

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly identifies the main change: adding ASIO support to NIXL.
Description check	✅ Passed	The description accurately covers ASIO support, conditional ETCD use, testing, and the new NIXL scenario.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch ipod/nixl-asio

_{Comment @coderabbitai help to get the list of available commands.}

podkidyshev · 2026-08-11T13:50:17Z

            assert cmd_args.total_buffer_size == expected_total_buffer_size


-def test_gen_etcd_srun_command(nixl_bench_tr: TestRun, slurm_system: SlurmSystem):


a few tests have been removed as they duplicate the same code tested by the central e2e test

coderabbitai

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

src/cloudai/workloads/nixl_bench/nixl_bench.py (1)
37-79: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Reject empty ETCD endpoints for non-storage backends.

When runtime_type is ETCD and etcd_endpoints is empty, uses_etcd returns False without checking backend. A UCX benchmark can therefore launch two processes without ETCD or an ETCD endpoint. Add model-level validation and a regression test that rejects UCX with empty etcd_endpoints.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/cloudai/workloads/nixl_bench/nixl_bench.py` around lines 37 - 79, Update
NIXLBenchCmdArgs model validation to reject configurations with backend UCX,
runtime_type ETCD, and empty etcd_endpoints, while preserving valid managed-ETCD
and ASIO configurations. Add a regression test covering this invalid combination
and asserting model validation fails; use the existing NIXLBenchCmdArgs and
NIXLBenchTestDefinition symbols.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@src/cloudai/workloads/nixl_bench/nixl_bench.py`:
- Around line 37-79: Update NIXLBenchCmdArgs model validation to reject
configurations with backend UCX, runtime_type ETCD, and empty etcd_endpoints,
while preserving valid managed-ETCD and ASIO configurations. Add a regression
test covering this invalid combination and asserting model validation fails; use
the existing NIXLBenchCmdArgs and NIXLBenchTestDefinition symbols.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: d255cc06-fd0f-42b9-a3d4-f9055087603c

📥 Commits

Reviewing files that changed from the base of the PR and between 6b7ba0d and 1fe0be1.

📒 Files selected for processing (9)

conf/experimental/test/nixl-bench-posix.toml
conf/experimental/test/nixl-bench-ucx.toml
conf/experimental/test_scenario/nixl-bench-network.toml
conf/experimental/test_scenario/nixl-bench-storage.toml
doc/workloads/nixl_bench.rst
src/cloudai/workloads/common/nixl.py
src/cloudai/workloads/nixl_bench/nixl_bench.py
src/cloudai/workloads/nixl_bench/slurm_command_gen_strategy.py
tests/workloads/nixl_bench/test_command_gen_strategy_slurm.py

coderabbitai

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (2)

src/cloudai/workloads/nixl_bench/nixl_bench.py (2)
39-42: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Do not serialize ASIO-only fields for the null runtime.

When runtime_type is unset, the branch at Line 78 removes only etcd_endpoints. It leaves asio_address and asio_port in cmd_args. The Slurm generator serializes every entry from cmd_args_dict, so a null-runtime command can receive ASIO arguments without --runtime_type=ASIO. This can break storage-only runs or apply ASIO settings to the wrong runtime.

Remove the ASIO fields unless self.uses_asio is true. Add a regression test for the unset-runtime path.
Proposed fix
         else:
             # ASIO performs direct peer-to-peer coordination and does not use ETCD endpoints.
-            cmd_args.pop("etcd_endpoints")
+            cmd_args.pop("etcd_endpoints", None)
+            if not self.uses_asio:
+                cmd_args.pop("asio_address", None)
+                cmd_args.pop("asio_port", None)
Also applies to: 78-81
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/cloudai/workloads/nixl_bench/nixl_bench.py` around lines 39 - 42, Update
the command-argument construction around the runtime handling branch to remove
both asio_address and asio_port whenever self.uses_asio is false, while
preserving them for ASIO runtimes. Add a regression test covering an unset/null
runtime and verify the generated cmd_args_dict excludes all ASIO-only fields.
90-93: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Fallback when nixlbench.csv is empty.

metric_observations reads every existing CSV. A zero-byte CSV can raise during parsing, and a header-only CSV returns no observations even when stdout.txt contains valid rows. Use the CSV only when it contains data; otherwise fall back to extract_nixlbench_data(stdout.txt).
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/cloudai/workloads/nixl_bench/nixl_bench.py` around lines 90 - 93, Update
metric_observations to use nixlbench.csv only when the file contains usable
data; handle zero-byte or header-only CSVs without raising and fall back to
extract_nixlbench_data(stdout.txt) when no observations are present.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@src/cloudai/workloads/nixl_bench/nixl_bench.py`:
- Around line 39-42: Update the command-argument construction around the runtime
handling branch to remove both asio_address and asio_port whenever
self.uses_asio is false, while preserving them for ASIO runtimes. Add a
regression test covering an unset/null runtime and verify the generated
cmd_args_dict excludes all ASIO-only fields.
- Around line 90-93: Update metric_observations to use nixlbench.csv only when
the file contains usable data; handle zero-byte or header-only CSVs without
raising and fall back to extract_nixlbench_data(stdout.txt) when no observations
are present.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: be3ceee8-9fb3-44f8-a126-e44116b1b176

📥 Commits

Reviewing files that changed from the base of the PR and between 1fe0be1 and 08fd99c.

📒 Files selected for processing (1)

src/cloudai/workloads/nixl_bench/nixl_bench.py

podkidyshev · 2026-08-11T15:18:23Z

@rutayan-nv could you please take a look? (it's not urgent)

podkidyshev added 2 commits August 10, 2026 22:19

nixl asio support

2249665

fix asio startup

8e4dfa9

podkidyshev self-assigned this Aug 11, 2026

podkidyshev added the enhancement improvements/extension to an existing feature label Aug 11, 2026

podkidyshev added 4 commits August 11, 2026 13:31

code cleanup

7dacb01

restore backwards compatibility

9f3775b

remove redundant case handling

2aee117

rename prop

1676f2a

podkidyshev commented Aug 11, 2026

View reviewed changes

podkidyshev added 2 commits August 11, 2026 16:34

updated nixl configs

9bf13b3

simplify config

1fe0be1

podkidyshev marked this pull request as ready for review August 11, 2026 14:45

podkidyshev requested review from jj10306, rutayan-nv and srivatsankrishnan as code owners August 11, 2026 14:45

Merge branch 'main' into ipod/nixl-asio

08fd99c

coderabbitai Bot reviewed Aug 11, 2026

View reviewed changes

fix copyright headers

9eafbf5

Navigation Menu

Uh oh!

[NIXL] ASIO support - #996

[NIXL] ASIO support#996
podkidyshev wants to merge 10 commits into
mainfrom
ipod/nixl-asio

podkidyshev commented Aug 11, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Aug 11, 2026 •

edited

Loading

Walkthrough

Changes

Uh oh!

podkidyshev Aug 11, 2026

Uh oh!

coderabbitai Bot left a comment

Uh oh!

coderabbitai Bot left a comment

Uh oh!

podkidyshev commented Aug 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

		assert cmd_args.total_buffer_size == expected_total_buffer_size


		def test_gen_etcd_srun_command(nixl_bench_tr: TestRun, slurm_system: SlurmSystem):

Uh oh!

Conversation

podkidyshev commented Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test Plan

Additional Notes

Uh oh!

coderabbitai Bot commented Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

podkidyshev Aug 11, 2026

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

podkidyshev commented Aug 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

podkidyshev commented Aug 11, 2026 •

edited

Loading

coderabbitai Bot commented Aug 11, 2026 •

edited

Loading