Skip to content

Fix TP warnings: torch 2.13 collective renames + init_process_group device_id - #197

Open
vasilevklart wants to merge 1 commit into
mstar-project:mainfrom
vasilevklart:fix/tp-deprecation-warnings
Open

Fix TP warnings: torch 2.13 collective renames + init_process_group device_id#197
vasilevklart wants to merge 1 commit into
mstar-project:mainfrom
vasilevklart:fix/tp-deprecation-warnings

Conversation

@vasilevklart

Copy link
Copy Markdown

Closes #166.

What

Two warning sources in mstar/distributed/communication.py on every TP/SP run:

  1. torch 2.13 renamed the single-tensor collectives — all_gather_into_tensorall_gather_single — and the old name emits a FutureWarning on each call. The same rename batch also hit reduce_scatter_tensorreduce_scatter_single (used by CommGroup.reduce_scatter), which Minor: Warnings related to TP #166 doesn't mention, so this PR covers both.
  2. init_process_group was called without device_id, so every barrier() warns that it has to guess the device from the current context.

How

  • Resolve the collective names once at module level with getattr(dist, "<new name>", <old name>)pyproject.toml allows torch back to 2.9.1, where the new names don't exist. The new APIs are exact signature drop-ins for the old ones.
  • Pass device_id=torch.device("cuda", self.global_rank) to init_process_group — the same rank→device mapping init_dist already applies via torch.cuda.set_device. Side effect worth noting: NCCL now sets up communicators eagerly at startup instead of lazily at the first collective.

Testing

The renames are validated by driving the real CommGroup methods on a 2-rank CPU gloo group (torch 2.13.0):

  • Before the patch, CommGroup.all_gather / reduce_scatter emit both FutureWarnings; after, they are silent.
  • Outputs are bit-identical (torch.equal) between the deprecated and replacement APIs on both ranks, for both collectives.
  • The torch-2.9 floor was simulated by deleting the new names before import: the getattr fallback engages (deprecation warnings reappear, as expected on old torch) and results stay correct.
  • ruff check . passes.

No CUDA machine was available for this change, so the device_id/barrier half is verified by inspection only (the ordinal matches the existing set_device call); a smoke run on a GPU box with TP ≥ 2 would be a welcome double-check.

torch 2.13 deprecated all_gather_into_tensor / reduce_scatter_tensor in
favor of all_gather_single / reduce_scatter_single; resolve the names
once at module level with a getattr fallback so the torch>=2.9.1 floor
keeps working. Also pass device_id to init_process_group so barrier()
no longer warns about guessing the device.

Closes mstar-project#166
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Minor: Warnings related to TP

1 participant