Skip to content

[fix] MD-TRT: handle c10::Error in bind_nccl_comm() probe loop - #4465

Open
apbose wants to merge 1 commit into
mainfrom
abose/fix-resolve-process-group
Open

[fix] MD-TRT: handle c10::Error in bind_nccl_comm() probe loop#4465
apbose wants to merge 1 commit into
mainfrom
abose/fix-resolve-process-group

Conversation

@apbose

@apbose apbose commented Aug 6, 2026

Copy link
Copy Markdown
Collaborator

In PyTorch >= 2.13, c10d::resolve_process_group() throws c10::Error when a group name is not registered (previously returned nullptr). The probe loop in bind_nccl_comm() had no try/catch, so it crashed on i=1 even when only group "0" existed, breaking the save→load path for distributed TRT engines.

Fix:

  • Wrap resolve_process_group() in try/catch(c10::Error) with continue (not break) so gaps from new_group(use_local_synchronization=True) don't stop the probe from finding numeric groups beyond the gap.
  • Apply the same treatment to the soft-return call site at the end of bind_nccl_comm() (group_name resolved but not yet registered).
  • Correct and expand comments to document the PyTorch numeric naming scheme and the gap scenario.

Tests: add tests/py/dynamo/distributed/test_bind_nccl_probe.py with:

  • TestBindNcclProbeDetection: single group auto-resolve, local_sync gap, multiple groups deferral
  • TestBindNcclProbeBinding: explicit world pin, subgroup pin, auto-resolve
  • TestBindNcclProbeE2E: full save→load cycle regression test

Description

Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.

Fixes # (issue)

Type of change

Please delete options that are not relevant and/or add your own.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

Checklist:

  • My code follows the style guidelines of this project (You can use the linters)
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas and hacks
  • I have made corresponding changes to the documentation
  • I have added tests to verify my fix or my feature
  • New and existing unit tests pass locally with my changes
  • I have added the relevant labels to my PR in so that relevant reviewers are notified

In PyTorch >= 2.13, c10d::resolve_process_group() throws c10::Error when
a group name is not registered (previously returned nullptr). The probe
loop in bind_nccl_comm() had no try/catch, so it crashed on i=1 even
when only group "0" existed, breaking the save→load path for distributed
TRT engines.

Fix:
- Wrap resolve_process_group() in try/catch(c10::Error) with continue
  (not break) so gaps from new_group(use_local_synchronization=True)
  don't stop the probe from finding numeric groups beyond the gap.
- Apply the same treatment to the soft-return call site at the end of
  bind_nccl_comm() (group_name resolved but not yet registered).
- Correct and expand comments to document the PyTorch numeric naming
  scheme and the gap scenario.

Tests: add tests/py/dynamo/distributed/test_bind_nccl_probe.py with:
- TestBindNcclProbeDetection: single group auto-resolve, local_sync gap,
  multiple groups deferral
- TestBindNcclProbeBinding: explicit world pin, subgroup pin, auto-resolve
- TestBindNcclProbeE2E: full save→load cycle regression test for PR #4428
@meta-cla meta-cla Bot added the cla signed label Aug 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant