Multicast e2e - #9912
Open
zeeshanlakhani wants to merge 38 commits into
Open
Conversation
zeeshanlakhani
force-pushed
the
multicast-e2e
branch
from
February 24, 2026 12:39
55ae07d to
a34381e
Compare
zeeshanlakhani
force-pushed
the
multicast-e2e
branch
4 times, most recently
from
March 6, 2026 01:46
52ad8d8 to
cc63562
Compare
zeeshanlakhani
force-pushed
the
multicast-e2e
branch
from
March 11, 2026 02:35
6949be9 to
29a7f10
Compare
zeeshanlakhani
force-pushed
the
multicast-e2e
branch
from
March 11, 2026 08:42
29a7f10 to
a0eecb4
Compare
zeeshanlakhani
force-pushed
the
multicast-e2e
branch
from
March 16, 2026 18:45
c16e309 to
9dfbab0
Compare
This PR adds omdb commands to inspect multicast state: - `omdb db multicast groups` - list multicast groups with optional state and pool name filters - `omdb db multicast members` - list group members with filters for group-id, group-name, group-ip, state, and sled-id - `omdb db multicast info` - show detailed info for a specific group - `omdb db multicast pools` - list multicast IP pools We also include: - Background task status display for multicast_reconciler - Integration tests for all multicast omdb commands
zeeshanlakhani
force-pushed
the
multicast-e2e
branch
8 times, most recently
from
April 21, 2026 01:32
4ed3d4e to
706ba96
Compare
zeeshanlakhani
force-pushed
the
multicast-e2e
branch
2 times, most recently
from
April 22, 2026 05:19
a0fcb56 to
f3754a5
Compare
zeeshanlakhani
force-pushed
the
multicast-e2e
branch
from
April 22, 2026 07:39
f3754a5 to
7df8871
Compare
zeeshanlakhani
force-pushed
the
multicast-e2e
branch
from
May 11, 2026 01:43
4b0859a to
3cf70c4
Compare
zeeshanlakhani
force-pushed
the
multicast-e2e
branch
from
July 15, 2026 14:28
d507943 to
fdfccba
Compare
Also carries the tools script fixes from #10825 (opte override p5p publisher and gcc-runtime version parse), so the install scripts work on this branch before that PR lands on main.
zeeshanlakhani
force-pushed
the
multicast-e2e
branch
from
July 15, 2026 14:52
fdfccba to
8fd9ce0
Compare
zeeshanlakhani
marked this pull request as ready for review
July 15, 2026 15:15
Collaborator
Author
|
We're using a non-mainline opte version, hence why that CI job (purposely) fails. |
This merges main to pick up the OPTE p5p publisher fixes (#10825), and adds [multicast] enabled = true to the single-sled and multi-sled Nexus SMF config partials so deployments start with multicast switch programming and the reconciler active.
zeeshanlakhani
force-pushed
the
multicast-e2e
branch
3 times, most recently
from
August 7, 2026 06:26
913a625 to
3fc58bf
Compare
Beyond the merge itself: - Tighten IPv6 multicast admission to the RFC 7346 usable-scope allowlist (admin-local, site-local, organization-local, global). This matching validation that landed in dendrite and thundermuffin. - Add API version 2026080400 (MULTICAST_SSM_EXAMPLE_DOCS) for the SSM example-address doc updates. - Bump multicast-tracking pins: dendrite 0d48b5a3, maghemite e4d24324, propolis df5fd0a3, thundermuffin 486559bc, sidecar-lite 461cbe19. - Drop the oxlog [patch] entry and the ls-apis crucible-pantry filter rule, as maghemite was updated.
zeeshanlakhani
force-pushed
the
multicast-e2e
branch
from
August 7, 2026 06:50
3fc58bf to
a793ae6
Compare
An instance update whose multicast specs contain duplicates would be rejected only after resolution had run, and resolution implicitly creates missing groups as part of the lifecycle. The rejected request stranded a memberless group in a "Creating" state that implicit lifecycle reaping would never collect, since collection only triggers on last-member-leave. Instead, we now reject identical identifiers before resolution. For aliases that remain past that check (a group's IP alongside its generated name), the resolver reports whether it created the group, and a rejected request rolls back only its own creations via `multicast_group_mark_removal_if_empty`. Ownership comes from the resolver itself, since a creation race can return a group another request created.
zeeshanlakhani
force-pushed
the
multicast-e2e
branch
from
August 12, 2026 07:39
9cd3cab to
cc983e9
Compare
…create We now split the multicast portion of instance reconfiguration into plan, apply, and compensate phases. Planning resolves and deduplicates the requested groups and drops any implicitly created groups when resolution rejects the request, before any observable side effect. Updates whose multicast snapshot no longer matches current membership are rejected with a conflict rather than overwriting concurrent joins and leaves. Implicit creations that remain after planning and end up memberless after a later failure are left to the reconciler's empty-group and orphan passes, which now share a minimum-age grace so a just-created group is not reaped while its first join is still in flight. A memberless group in "Creating" is never activated: the reconciler lists members first, reaps only past the grace window, and declines if a member attached since the listing. Compensation now reverses only the changes a request made itself. The datastore attach returns an `AttachMemberResult` carrying an inserted flag that distinguishes a fresh insert from an upsert hitting an existing active row, so a failed request no longer detaches a membership that a concurrent request established. Detaches are recorded the same way, from the datastore's row-affected result. Retried saga actions recognize groups and members from their earlier invocations and undo only what they own. That recovery spans the orphan pass: a reaped group that never reached the dataplane is restored with a fresh grace window, and a hard-deleted row frees the stable ID for re-creation. Only a soft-deleted row that had entered the dataplane blocks retries, surfaced as a retryable 503 until cleanup hard-deletes it. Deletion processing now skips dataplane teardown for a group whose underlay link was never written. The link is set before the DPD saga launches and is never cleared, so its absence proves the group owes no switch or sled cleanup. This also closes a race in which a slow delete pass could wipe fresh DPD state from a group resurrected and re-activated under the same tag, since resurrection requires the link to be absent.
Tag and source updates read the underlay group and had written the read member list straight back. `multicast_group_update_underlay` is a full-list replace whose only concurrency control is the per-group tag, so that echo-write could silently revert a member add or remove that landed between the read and the write. `update_groups` now reads the underlay group only to obtain the tag authorizing the external update, creating the group (memberless) when absent, and never writing the underlay member list. The member add and remove paths remain the list's only writers. This also removes an underlay write the planned member-writer transition would otherwise have to delete.
zeeshanlakhani
force-pushed
the
multicast-e2e
branch
from
August 13, 2026 00:36
cc983e9 to
8fc8591
Compare
Group drift detection now returns a configuration only when every switch reports the same one. A group absent from some switches or diverging across them yields `None`, which drives the reconciler into the update path that rewrites the group on every switch. Agreement, here, is semantic rather than structural: `external_group_id` is a switch-local allocation and is ignored, and sources compare as sets since DPD does not guarantee read-back ordering. Comparing either would flag permanent drift and rewrite the group on every pass.
A join that omits `source_ips` means "keep my filter". Resolve the group by lookup first and, when the instance already holds a membership in it, skip the validating resolver that would otherwise reject the omitted sources on an SSM group. A lookup that finds nothing falls through to the resolver, leaving implicit creation and its validation unchanged. This extends the SSM validation test with repeat joins via ID, name, and IP, asserting the membership ID and stored source filter remain intact.
A failed undo action permanently strands the saga in Steno in an unrecoverable state, and the rollback could fail at client construction or group removal. Now, we log failures and return success instead: leftover DPD state is keyed by the group tag, so the next reconciler pass rewrites it while the group is still "Creating" or removes it by tag once it is in a "Deleting" state. Partial `create_groups` failures need no rollback whatsoever. Steno does not run a failed node's own undo, and that is the desired behavior here: the group stays "Creating", the reconciler restarts the saga, and the dataplane client's CONFLICT paths fill in missing switches while leaving configured ones forwarding. Attribute drift on configured switches heals in the "Active" drift check, and the `create_groups` error log no longer claims a rollback will run.
The instance-update saga's `sled_id` branch persisted the new sled for multicast members but did not activate the reconciler, leaving traffic pointed at the old sled until the next periodic pass. Activate it the way the other member-mutating branches already do.
The "Deleting" reconciler pass tears down the dataplane by tag and hard-deletes the underlay group, members, and external group rows in the same pass, so no row ever reaches "Deleted" and no code path has ever written it. Deletion visibility is already carried by `time_deleted`, set when the group enters "Deleting", so a terminal state adds nothing. Remove the variant, its dead reconciler branch, and the omdb filter arm. The 'deleted' enum value shipped in the multicast-group-support migration, so it cannot be edited in place. The drop-multicast-deleted-state migration removes it with the temporary-type sequence from the schema README. Dropping the state column requires dropping the indexes that key on it, so the two state indexes are dropped ahead of the swap and recreated at the end.
Crash recovery can re-execute the saga after the group has already
reached "Active". Rejecting that state would prevent replay of already-
applied DPD state, so the fetch action now allows "Creating" and
"Active". Every downstream action tolerates re-execution, so replay
against an "Active" group rewrites the same DPD configuration and
re-asserts "Active".
Extends the state-validation test to re-run the saga against an
"Active" group and to reject a terminal state ("Deleting").
…89, drop OPTE override) Main took schema 288, so drop-multicast-deleted-state moves to 289. OPTE 0.41.506 is now published with the multicast work included, so OPTE_COMMIT in tools/opte_version_override is cleared to match main.
zeeshanlakhani
force-pushed
the
multicast-e2e
branch
from
August 17, 2026 10:32
8c43e15 to
b272b8d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is the integration branch for multicast support across the stack. A lot of the Nexus-side multicast work has already landed on main through earlier PRs. This consists the cross-repo dep pinning for the in-flight revisions of the other repos that multicast depends on, OMDB tooling, and test updates.
Cross-repo pins
ab6c1a43(defaults the dpdmulticastfeature on and wires the SoftNPU backend for multicast group programming): dendrite#2240509f500(DDM underlay multicast route origination/import): maghemite#696f4433191(xde multicast delivery), viatools/opte_version_override: opte#1012cd0e2618(viona receive filtering): propolis#1093284c6830/ sidecar-lite6f3311e8(dev virtual hardware, bitmap-based replication): softnpu#183,sidecar-lite#152
fda704ee(UDP multicast send/recv, merged): thundermuffin#13omdb support
multicast_reconcilertask status, including the disabled case.Test and schema updates
allow_tables_to_appear_in_same_query!for the multicast tables, needed by the omdb joins.object_put_upserttest helper for PUT endpoints that return 201.