Skip to content

[wip] fix(iroh): survive home relay changes on relay-only connections - #4435

Draft
Frando wants to merge 5 commits into
tests/patchbay-relayfrom
fix/relay-failover
Draft

[wip] fix(iroh): survive home relay changes on relay-only connections#4435
Frando wants to merge 5 commits into
tests/patchbay-relayfrom
fix/relay-failover

Conversation

@Frando

@Frando Frando commented Jul 22, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Description

The first commit adds a patchbay test that fails on main: On a relay-only connection with no working IP path, we destroy connectivity to the relay. The test fails on main. However, this doesn't have to be, we can make the connection survive: Breaking the relay connection could trigger a net report, chose a new home relay, and handle the new relay URL to the client over addr lookup, so that it can open a new path over the new relay.

This PR explores how to we can do this.

In its current form, it does this:

  • a) When the home relay connection dies, trigger a net report, so that we chose a new home relay and publish it in addr lookup
  • b) When a relay connection dies and reconnect is not successful, inform the remote state actors about this. If we have any client-side connection where the now-dead relay path is the only path, trigger addr lookup (and retry with backoff) to see if the remote published a new relay URL (which it now does thanks to a) above).
  • If we discover a new relay URL, establish a new path over this relay URL for all connections that used the previous relay URL, and close the previous relay path (i.e. do as with IP path: close all but one relay path).
  • Use fix(proto): On abandoned paths declare packets as lost after some time noq#770 so that data on the now-dead relay path is retransmitted soonish. Without this noq change, application data sent over the now-dead relay path would stay in that path's send queue for a long time, because its PTO (which eventually triggers draining and declaring all packets lost after) becomes hugely inflated: After the path dies, no data flows for a while, until our new relay path is established, which then delivers acks for the last delivered packets on the now-dead path. Because discovering and establishing the new relay path takes several seconds, the previous path's RTT is inflated to a few seconds, making its PTO be inflated too, which delays the draining and declaring its packets lost to (in the patchbay test measured to be more than 30 seconds). The linked noq change fixes this because packets are now declared lost after 2*PTO at abandon time, when the PTO is not yet inflated.

Depends on n0-computer/noq#770
Includes #4444

Breaking Changes

Notes & open questions

Draft. The fix was mostly done by an AI tool and still needs to be reviewed.

Change checklist

  • Self-review.
  • Documentation updates following the style guide, if relevant.
  • Tests if relevant.
  • All breaking changes documented.
  • This PR was created by a human that thought critically about the
    proposed change and wrote an as clear and concise description as
    they could.
  • This PR isn't slop, and is carefully crafted to do have the
    intented effect.

@github-actions

github-actions Bot commented Jul 22, 2026 •
edited
Loading

Copy link
Copy Markdown

Documentation for this PR has been generated and is available at: https://n0-computer.github.io/iroh/pr/4435/docs/iroh/

Last updated: 2026-07-27T08:21:38Z

@github-actions

github-actions Bot commented Jul 22, 2026 •
edited
Loading

Copy link
Copy Markdown

Netsim report & logs for this PR have been generated and is available at: LOGS
This report will remain available for 3 days.

Last updated for commit: 84ca5c8

@n0bot n0bot Bot added this to iroh Jul 22, 2026
@github-project-automation github-project-automation Bot moved this to 🚑 Needs Triage in iroh Jul 22, 2026
@Frando Frando changed the title fix(iroh): survive home relay changes on relay-only connections [wip] fix(iroh): survive home relay changes on relay-only connections Jul 23, 2026
@Frando
test(iroh): patchbay test for relay failover via address lookup
4445d78 
relay_teardown_failover gives the peers disjoint relay maps: the server
holds a fast and a slow relay (homing on the fast one), the client its
own unrelated relay. The connection runs over the server's home relay,
which is torn down mid-connection. Address lookup runs against an
iroh-dns-server in the lab (new linux-only dev-dependency): the server
publishes over pkarr HTTP, the client resolves the _iroh TXT records on
port 53, the only UDP port its firewall allows. Util grows a named
relay spawner and the dns-server spawner.

The test needs the failover fixes in the next commits to pass.
@Frando
Frando force-pushed the fix/relay-failover branch 2 times, most recently from a0f8a26 to dc90a4e Compare July 24, 2026 12:12
Frando added 4 commits July 25, 2026 14:17
@Frando
fix(iroh): force a full net report when the home relay is lost
855acf8 
When the home relay connection drops, incremental net reports run only
QAD probes. If UDP is unavailable those measure nothing, so the report
returns no relay latencies and no other configured relay is promoted to
home; the endpoint retries the dead relay for up to the full-report
interval (5 minutes) instead of failing over.

The socket actor now watches the home relay connection. On loss it
schedules a full net report, deferred by two seconds so a brief
reconnect does not trigger one, and rate-limited so a still-down relay
does not force back-to-back reports. A full report re-runs the HTTPS
probes and lets a reachable relay be promoted.
@Frando
fix(iroh): answer HasEndpointRoute during relay reconnect backoff
942dea4 
active_relay_handle_for_endpoint queries every tracked active relay via
HasEndpointRoute before creating a new relay connection. The outer
reconnect backoff loop slept via a bare time::sleep that never polled
prio_inbox, so a query sent to a relay mid-backoff (e.g. one that just
died) sat unanswered until that unrelated backoff elapsed, directly
gating unrelated new relay connections. sleep_backoff now answers
HasEndpointRoute(false) immediately while still counting down the
delay, mirroring run_dialing's existing pattern. Path establishment
after a relay teardown is now consistently ~1-1.5s instead of up to
15s+ in traced runs.
@Frando
fix(iroh): re-resolve the remote when the relay carrying it is lost
34ff7d2 
The relay path of a connection runs over the remote's home relay; the
peers do not share relay maps. When that relay dies, the remote fails
over its home relay and republishes its endpoint info, but the client
kept retrying the dead relay: nothing re-resolved the remote or opened
a path over its new home relay. Opening a path over our own home relay
cannot help, since the remote is not connected to it.

Detection uses the relay transport's own knowledge: each
ActiveRelayActor broadcasts RelayConnEvent::Lost/Restored for its
relay. Lost is debounced to skip spurious disconnects: an established
connection dropping retries immediately without emitting anything, so
the event only fires once a reconnect attempt (several round trips of
connect+handshake) has also failed. On Lost, a RemoteStateActor whose
connection has no way around that relay forces an address lookup via
trigger_address_lookup(true): not running starts it immediately;
already running queues a retry for when the current attempt finishes.
The retry repeats with backoff (500ms up to 5s) until a replacement
path establishes, the relay connection is restored, or the connection
closes.

Selection also keys on the same transport state: lost relays are
passed into the path selection context and their paths marked as
transport-down (PathSelectionData::transport_down). The selector only
uses such a path as a last resort, so the replacement relay path wins
selection despite the dead path's better (stale) RTT, after which the
dead path is closed as a redundant relay path (apply_selected_path,
generalized from the existing IP-only redundant-path closing).

For the case without any transport signal - the remote lost *its*
relay connection while ours is fine - the path idles out (keep-alives
stop being acknowledged) and noq abandons it; a connection whose last
path is abandoned also forces the lookup.

Uses noq from the Frando/abandon-retransmit branch (PR pending
upstream): closing the dead relay path declares its in-flight data
lost so it retransmits over the replacement path, instead of
stalling until the PATH_ABANDON round trip completes.
@Frando
refactor(iroh): Option<Receiver> instead of a separate closed flag
f6fa4b4 
Matches the existing maybe_next pattern for the address lookup stream:
the receiver is dropped once the channel reports closed, making the
closed state unrepresentable instead of tracked alongside.
@Frando Frando mentioned this pull request Jul 27, 2026
Draft
7 tasks
@Frando
Frando force-pushed the fix/relay-failover branch from dc90a4e to f6fa4b4 Compare July 27, 2026 08:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

iroh
Status: 🚑 Needs Triage

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@Frando