Implement graceful versiond host evacuation and replacement - #1517
Closed
snevolin wants to merge 77 commits into
Closed
Implement graceful versiond host evacuation and replacement#1517snevolin wants to merge 77 commits into
snevolin wants to merge 77 commits into
Conversation
Track B: graceful evacuation, replacement, addition and permanent decommission of a whole versiond host. - table-driven host lifecycle and full-response admission leases in versiond - one absolute, configurable shutdown budget for proxy drain, child drain, HTTP shutdown, escalation and child reap - table-driven child-generation, supervised-process and devshardd lifecycle machines - persistent router membership with immutable membership IDs and an explicit one-host-at-a-time transfer - forward-reconciling router control plane with revisioned nginx config projections, durable completion receipts and an audit outbox - local gonka-routerctl and resumable SSH-based gonka-hostctl workflows - full-stack coverage for a sticky long-running SSE request, evacuation, replacement, decommission and re-addition Replay of sn/versiond-host-evacuation (3f68fde..2064415, 40 commits) onto upgrade-v0.2.15. The original history could not be rebased: two upstream backmerges carried manual conflict resolutions that a flattening rebase drops. The resulting tree is byte-identical to the source branch.
0.2.14 is released, so its release guide is restored to the base state. Add devshard/docs/release-0.2.15-v5.md following the v4 guide structure and carry the graceful-shutdown section and rollout checklist item over verbatim.
Move the Docker host evacuation acceptance job into a dedicated workflow with path filters for Track B code, devshardd lifecycle code, the test harness, and its build inputs. Cancel superseded runs for the same pull request so unrelated testenv changes and stale pushes do not consume the 40-minute runner budget.
Stop triggering the 40-minute acceptance test for every root Makefile change. The shared Testermint workflow already exercises the devshardd-build target, while the dedicated workflow retains triggers for Track B code, its test harness, and direct build metadata.
snevolin
force-pushed
the
sn/versiond-host-evacuation-0.2.15
branch
from
July 29, 2026 02:52
d5825e6 to
1927922
Compare
versiond-router had no Go code and no persisted state before this work, so router state schema 1, WAL schemas 1-4 and hostctl journal schemas 1-2 only ever existed inside this branch. No deployment wrote them. Remove the migration paths and pin every persisted format at schema 1: - delete router state and operation-journal migrations; decode the current schema directly and reject any other version - delete the legacy rollback recovery policy, the pre-image journal fields it needed, and the config restore it performed - delete the audit-to-receipt-index import and the ActionAgnostic/Conflict receipt fields that only that import could set - delete the hostctl journal migration and the host_idle resume alias Torn-tail audit repair stays: it protects the append path independently of the removed import.
Host evacuation was built around nginx OSS not being able to discover that a
versiond is draining: since it cannot health-check upstreams, something had to
tell it. That something was a 12.9k-line control plane — a durable router FSM,
membership IDs, transfer ownership, a WAL, a receipt index, an audit outbox, and
two SSH-driven CLIs — whose entire job was to move an upstream to `down` before
anyone sent SIGTERM.
Move both routers to HAProxy and let them observe instead:
* membership comes from DNS. VERSIOND_POOL_HOST / EDGE_API_POOL_HOST is one
name with an A record per running instance (a Compose network alias), read
through `server-template` + `resolvers`. Starting or stopping a container is
the whole operation; no config change, no reload.
* health comes from `GET /readyz` once a second. A host that is draining,
still converging, or cut off from the chain takes no traffic and rejoins on
its own.
versiond makes that safe with a new `announcing` state: on SIGTERM it fails
/readyz while still accepting for VERSIOND_DRAIN_ANNOUNCE (5s), so the router
removes it before admission closes. Readiness moved to the traffic listener,
which drops the loopback admin listener entirely and is what a Kubernetes
readinessProbe will consume unchanged.
Readiness also had to be relaxed: `Converged` now latches once every desired
version has run. Without that, a routine same-name SHA bump — published to every
host at once — would un-converge the whole pool simultaneously and evict it.
HA storage safety gains a startup half: with GONKA_HA set, devshardd refuses to
boot on storage a sibling cannot see, instead of booting and failing every
HA-marked request. The per-request Devshard-Ha guard stays, because a partial
rollout can leave a process that started before the deployment became HA.
edge-api gets the same treatment: a cached /readyz that reports chain
reachability, so an instance that loses the chain leaves the round-robin
rotation rather than answering.
What remains of the control plane is `gonka-drain`, an 80-line shell guard over
the HAProxy Runtime API for quiescing a host without stopping it. It addresses
hosts by container name or IP and refuses to drain the last one still serving.
Failover discipline is preserved from the nginx config: retry on connect
failure, empty response, and upstream 502; never on 503, which is what a
draining host and the storage guard answer; and `disable-l7-retry` for
non-idempotent methods so an inference is never executed twice.
Net effect on the PR: +6.0k/-0.8k across 75 files, down from +18k.
Verified end-to-end against real HAProxy 3.0 containers: sticky hashing by
escrow, legacy version pinning, /devshard prefix handling, Devshard-Ha
stamping, health-based eviction and re-entry, DNS-driven join and removal,
the last-host drain guard, and edge-api round-robin with health eviction.
`make test-render` renders every supported shape and validates each result with
the HAProxy the routers ship.
Adding /readyz to edge-api put it behind an active health check without giving
it anything to announce: SIGTERM went straight into Echo.Shutdown with a
hardcoded 10s, /readyz kept answering 200 while the chain was reachable, and no
Compose file set stop_grace_period. Replacing an instance therefore cut every
query still running at ten seconds, and the router only found out afterwards.
Mirror the versiond sequence:
* BeginDrain latches /readyz to 503 with reason "draining", checked before the
chain probe so it does not wait out the readiness cache;
* keep serving for EDGE_API_DRAIN_ANNOUNCE (5s) so the router's 1s check
removes the instance before it stops accepting, with a second signal cutting
the wait short;
* then Shutdown under EDGE_API_SHUTDOWN_BUDGET (2m, matching the router's
default read timeout — the process should wait exactly as long as the hop in
front is still willing to wait), and Close with a diagnostic if it expires.
Malformed durations now fail at boot rather than silently falling back to a
default that is only wrong during an outage.
Every Compose file that runs edge-api gets stop_grace_period: 3m, including the
single-instance base. Without it Docker's 10s default would SIGKILL the process
in the middle of the drain it just learned to do.
Verified against the built binary with a chain endpoint that accepts and never
answers: /readyz reports "draining" at t+1s while /healthz still serves; a query
in flight survives past the old 10s cutoff and keeps running to t+28s (announce
plus budget); at budget expiry the process closes it and logs
"graceful shutdown did not finish" instead of waiting for SIGKILL.
Three gaps in the previous commit, from review:
The single-instance Compose never passed the settings. Compose forwards only
what a service lists, so EDGE_API_DRAIN_ANNOUNCE and EDGE_API_SHUTDOWN_BUDGET in
config.env reached nothing and the built-in defaults were always used. Both are
now declared on the base edge-api service, and the announce window is in
config.env alongside the budget it pairs with.
A second signal could only cut the announce window short. Once Shutdown started,
nothing read the signal channel — and signal.Notify had already taken SIGTERM
away from the runtime, so an operator staring at a stuck two-minute drain had
nothing left but SIGKILL. Shutdown now runs in a goroutine while the caller
selects on {finished, budget expired, another signal}, and either failure mode
closes the remaining connections.
Escalation goes to http.Server.Close rather than echo.Close, which turned out to
matter: echo.Shutdown holds startupMutex for its entire run, so escalating
through Echo deadlocks against the very drain it is meant to interrupt. The
end-to-end test below is what found that — the fix was written first and hung.
The sequence itself had no test. It is now extracted as drainAndShutdown and
covered against a real server on a real port with a blocking handler:
* an in-flight request survives the drain and completes, while /readyz already
reports "draining" — both halves of the guarantee in one test, so either one
regressing fails CI;
* budget expiry ends the wait and says why;
* a signal during Shutdown forces the close (this is the deadlock guard);
* BeginDrain is called before Shutdown, not after.
Two follow-ups from review. The budget was only enforced through the error Shutdown chose to return, so it depended on Shutdown reaching the point where it looks at its context. A Shutdown blocked before that — on a lock, say, which is exactly how the echo.Close deadlock behaved — would have run past the budget unbounded. Watch ctx.Done directly alongside the result and the signal, so the ceiling holds whether or not the thing it bounds cooperates. Shutdown also reports a failed listener close the same way it reports a deadline, and wrapping both as "shutdown budget expired" would send the next operator to tune a setting that has nothing to do with it. Only context.DeadlineExceeded is now called a budget expiry; anything else keeps its own cause. Both guards were checked against a reverted implementation: without the ctx.Done branch the blocking-Shutdown test hangs to the test timeout, and without the cause check the listener-failure test fails.
A server's position on a consistent-hash ring defaults to its numeric slot id.
With server-template the slots are filled from a DNS answer whose order nothing
guarantees, so a plain router restart could hand the same hosts different slots —
and move every session at once. That is the one failure the sticky pool exists to
prevent, and it would have happened silently on a restart nobody thought of as
risky.
Measured with four backends and the same addresses in different slot orders,
default keying:
forward order: .5 .2 .2 .2 .4 .2 .5 .3
reversed order: .2 .5 .5 .5 .3 .5 .2 .4
Not one of the eight escrows stayed put. With hash-key addr the mapping is
identical under forward, reversed and shuffled slot order. hash-key is a server
keyword and is already available in the HAProxy 3.0 the routers ship, so no image
bump is needed.
Two guards, at the two costs they are worth:
* make test-render asserts the pool's servers carry hash-key addr, so removing
it fails on every PR;
* the evacuation acceptance test now restarts the router and requires the
escrow to come back to the same host, which is the behaviour rather than the
spelling.
Moving onto this router re-homes sessions once, since the ring is not the one
nginx computed. HA sessions recover from shared Postgres, so that costs a lookup.
Two problems with the previous commit's guard, from review. The acceptance test restarted the router and then kept talking to the port it had before. The harness publishes random host ports, and Docker hands out a new one when the container comes back, so the suite failed with connection refused — reproducibly, in CI. Endpoints are re-resolved after the restart now. That test also could not prove anything: a restart does not force DNS to answer in a different order, so the very regression it was written for would have sailed through it. It stays as a smoke test, with a comment saying so, and the proof moves somewhere it can be made deterministic. `make test-hash-ring` takes the hashing directives out of the rendered config, puts the same four addresses into the server slots in forward, reversed and shuffled order, and requires every escrow to reach the same address in all three. It needs no upstreams: HAProxy logs which server it selected even when the connection is refused, and that selection is the entire question. Checked in both directions. With `hash-key addr` all three orders agree; with it removed from the template the test fails and names the sessions that moved. The first attempt at that negative check passed by accident — the extraction matched `hash-key addr` inside a comment — so it now reads only the server-template line. test-render depends on it, so CI and `make -C devshard ci-testenv-unit` pick it up with no workflow change.
The rejoin check required observing the target DOWN right after start, which is a race the test loses whenever the child comes up quickly — reproducibly, per review. Waiting for a transient is no way to check that an unready host gets no traffic; the host has to be held unready. It now comes back pointed at an oracle that refuses connections. versiond boots, appears in DNS and stays there, but never learns what to run, so /readyz keeps failing for exactly as long as the test holds the barrier — no transient to catch. Lifting the barrier restores the real oracle and the host rejoins. Verified the barrier directly against a versiond container: alive with /healthz 200 and /readyz 503 at 6s and still at 16s, reconcile failing on every poll. It is a state, not a window. The barrier is deliberately scoped to one host, which is why PatchComposeEnvKey would not do — it rewrites the key in every service, and repointing the survivor too would just empty the pool. PatchComposeServiceEnv edits one service block and returns the value it replaced, so the restore is the file's own value rather than one reconstructed from config. It has its own unit test, including that a sibling service and an unrelated block with the same key are left alone. This also pins something worth pinning: one host that cannot converge does not make the pool unready. The survivor keeps its oracle, keeps serving, and the test asserts the session stays reachable on it throughout.
/readyz gated on Conditions.Degraded, which is set by any reconcile error — including "oracle fetch failed". Every versiond reads the same oracle, so that error does not arrive on one host: it arrives on all of them, within one poll interval of each other. An unreachable oracle or one bad archive would therefore have emptied the pool, and the router would have answered 503 for every version, while every child was still running and serving normally. That is the same correlated-failure shape the Converged latch already fixes, and it contradicts what the endpoint is for: readiness answers "can this host serve now", not "did the last control-plane poll succeed". Reconcile failures keep being reported through Degraded, /healthz and the logs, which is where a deployment problem belongs. The cost is named rather than hidden, in code and in the docs: a host that has served before and then fails to install a newly approved version stays in the pool, so requests for that one version fail on it instead of moving to a host that installed it. Fixing that properly needs per-version readiness, which one balancer health check cannot express. Should a condition turn up under which accepting traffic is genuinely unsafe, it gets its own typed condition — not a ride on the generic reconcile error. The acceptance-test barrier is unaffected and was re-checked against a real container: a host pointed at a dead oracle has never converged and has no child, so it stays 503 on /readyz — alive, /healthz 200, still 503 at 18s — with Available and Converged now carrying that on their own.
Two docs said a failed reconcile is reported "through the Degraded condition, in /healthz and in the logs". Only the last of those is true: /healthz serves health.StatusEntry — name, port, status, sha256, binary_version — and nothing else. Promising a field that is not there is worse than saying nothing, because an operator can build an alert on it and only find out during the outage it was meant to catch. Corrected to what actually happens: the failure is kept in versiond's internal Degraded condition and logged at ERROR, and /healthz is deliberately unchanged because that array is a contract existing clients parse. The gap is recorded as a follow-up rather than papered over. Reconcile failures have no machine-readable exposure today; that belongs in a metric, not bolted onto the legacy JSON, and versiond has no metrics endpoint of its own yet.
A new SHA that will not download reaches Degraded through the reconcile result, not through ReportReconcileError, so the existing regression test for an unreachable oracle did not cover it. Both triggers are fleet-wide — every host reads the same oracle and the same archive — and both must leave a serving host in the pool. The test asserts what the balancer consumes after a failed install: the old child is still running so Available holds, Converged is not retracted, and the failure is still reported as Degraded. That downloadAndSwap keeps the old child is already covered by TestDownloadAndSwap_NewChildNotReadyKeepsOldServing; this pins the conditions that follow from it.
The docs said a reconcile failure is logged at ERROR. An unreachable oracle is, but an empty version list sets the same Degraded condition and logs at WARN, so anyone grepping for the promised level would miss half the cases. Say only that it is logged.
A single readiness flag per host cannot say "I am missing one of several versions", so the choice was between letting a host serve a version it does not have, and evicting it entirely. Both are wrong, and the second is worse: the cause of a missing version is a shared archive, so it is missing on every host at once and the whole pool would leave over a version most traffic does not use. So stop asking the host-level question. versiond now answers /readyz?version=<v> from the same route table the proxy uses, and the router keeps one backend per version in VERSIOND_VERSIONS, health-checked with that version's own question. Same hosts in every backend; what differs is which of them pass. A host that cannot run v5 leaves v5's ring and keeps serving v4. Approving a new version becomes safe for the same reason: until a host has v6 running it is simply not in v6's pool, so v6 traffic goes only where it can be served and v4 carries on untouched. The per-version answer needs no convergence latch and no view of the desired set, which is what makes it precise — either a running child serves that version here or it does not. The host-level /readyz stays as the fallback for versions the router was not told about, so listing a version buys precision and is never a prerequisite for serving it. All three pool backends are rendered from one pool-backend.cfg.template, so the routing policy — hashing, hash-key addr, retry discipline, header handling — cannot drift between them. test-render asserts each rendered backend carries all five policy directives, which is the check that would catch a fragment edited for one pool and not the others. Verified end to end against the shipped image with two upstreams that disagree about which versions they serve: v4 spreads over both, v5 reaches only the host that has it, an undeclared version falls back to the host-level pool, and gonka-drain can inspect any of them. make test-version-routing pins exactly that, and fails when the version dispatch is removed.
Three holes in the previous commit, from review. gonka-drain drained one backend. HAProxy server state belongs to a backend/server pair, and a host now sits in every pool, so draining it from versiond_ha_pool left it serving every declared version — the command reported success and did almost nothing. It now plans across all pools, refuses if any of them would be emptied, applies, and rolls back if one apply fails, so a host is never left half out. Verified: draining one of two hosts removes it from v4 and v5 at once, and putting it back restores both. An undeclared version fell back to the host-level pool, which is the coarse check per-version pools exist to replace: the request reached whichever host the hash picked and 404'd there if that host lacked the version. So the claim that approving a new version was automatically safe was wrong, and the docs said so. While any version is declared, an undeclared one is now refused at the router with a 503 that names the setting to fix. Non-version paths like /healthz are exempt, and an empty VERSIOND_VERSIONS disables the mechanism entirely and keeps the previous behaviour. The two-phase rollout — declare, then approve — is documented as the procedure it is. Version names had two contracts. The router accepted only [A-Za-z0-9._-] and refused to *boot* on anything else, while the chain accepts any non-empty name and versiond only bars path separators. A legal approved version could therefore take the router down. The name is now taken as governance wrote it and only the HAProxy identifier is derived from it, with a loud failure if two names derive the same one — a silent collision would merge two versions' pools. test-version-routing covers the new refusal and that /healthz survives it; the render test covers name derivation and collisions.
Six problems, from review. The acceptance test failed on the drain refusal wording. The message and the assertion now say the same thing, and the message names the backend it is protecting. The strict guard called the first path segment a version, so it refused /metrics, /stats, /devshard/healthz and the session observability routes — all of which versiond serves without a version. It now classifies against that grammar (proxy.go: isVersionlessObsPath) on the canonical path, so the prefix form is covered too. Checked against each of those paths. gonka-drain skipped versiond_legacy, so draining the legacy owner reported success while legacy traffic kept arriving. Every backend the host appears in is now included, which makes the legacy owner undrainable — its backend has one server, and emptying it would fail every pinned version. That is the honest answer, and it is now the one the command gives. A legacy backend with nothing pinned to it is skipped, since no request can reach it. The last-server check counted the backend rather than the target: a host that serves v4 but not v5 could not be drained, because v5's pool had one server — which was not the host being drained. The guard now applies only where the target is itself taking traffic. Version names went through a lossy tr, so v5+cuda and v5-cuda derived the same backend, and '+' in the health-check query decodes to a space on the versiond side, leaving the host down forever. One grammar now applies — [A-Za-z0-9][A-Za-z0-9._-]* — used verbatim as backend name, query value and map key, with anything else refused at startup. Narrowing the chain's own validation to match is recorded as a follow-up. The rollback was not one: it applied the opposite action instead of restoring what was there, and two concurrent drains could each see a live peer and then leave none. Admin states are snapshotted per server and restored exactly, and the whole plan-and-apply is under a lock. Verified: concurrent drains leave one host serving, and the loser is told why. The documented rollout said "restart the router", which does not pick up an environment change, and recreating it cut live streams. The procedure is now `up -d --force-recreate`, and the router stops on SIGUSR1 — HAProxy's soft stop — so the outgoing container finishes its streams.
…prove Seven problems, from review. The acceptance test asked the host-level /readyz while waiting on a per-version pool; those do not turn 200 at the same moment. The harness was also flattening every backend into one list, so a wait settled on whichever backend printed last. RouterSlot now carries its backend, the waits name the pool they mean, and TryVersiondReady asks the same question the router asks. Replacing the router to declare a version refused every new connection from the moment the old container was told to stop until it finished its longest stream — minutes, up to stop_grace_period, not the "short gap" the docs claimed. Added gonka-reload: render, check, then SIGUSR2, which is HAProxy's master-worker reload. The listener never closes and established streams stay on the old worker. Measured over 1200 requests at 92/s, one reload cost a single 503, a window under about 10ms. Server states are handed to the new worker first, or it would re-probe from scratch — and would also forget which hosts an operator had drained. Version names are no longer restricted to what happens to be a valid HAProxy identifier, which the chain does not promise. Each of the three uses now gets a form that suits it: the map key is the name as written, because it is matched against the path segment; the health-check query is percent-encoded, because '+' there decodes to a space; the backend identifier gets a hash appended when the name is not already one. v5+cuda routes end to end. A name that cannot appear literally in a path segment is still refused, because the path would not match it — that residue is documented rather than papered over. gonka-drain matched slot names per backend, but a slot is local to its backend: versiond1 is a different host in each. Draining by slot could leave the legacy backend serving. The target is resolved to one address first and every backend is matched on that address alone. The versionless ACL was unanchored, so /metrics-v9 and /healthz-v9 slipped past the undeclared-version guard. Each alternative is anchored now, and both are refused while /metrics and /healthz still work. Duplicate detection matched with a regex, so 'v1.2' collided with 'v1x2'. It compares the map's first field exactly. gonka-drain read the legacy map from disk, which misses a pin added through the Runtime API. It reads the live map from HAProxy.
The last review's closing point is the one that matters: gonka-reload put a second source of truth back into the router. Four of its findings are the same finding — a reload re-rendered the config from the environment and so discarded runtime map pins, left drains applied in some pools and not others, resurrected stale drains from a state file on disk, and reported success before the new worker was serving. That is a control plane growing back, one convenience at a time, in the PR that deleted one. So the reload is gone, along with the server-state file it needed. Declaring a version changes the deployment and replaces the container, as any other router config change does. To keep a governance approval from needing that at all, the join overlay now declares a window — v4 through v8 — because a pool for a version nobody runs has no healthy members and costs nothing but its checks. The README says this plainly instead of offering an in-place path that cannot be made consistent. Separately, a real routing bug: a host was selectable the moment DNS mentioned it, before its first health check had run — a window straight onto a versiond that is still starting, which is what the acceptance test kept catching. HAProxy 3.1 added `init-state fully-down` for exactly this; the routers move to 3.2, still LTS, and every pool server now starts down and has to earn its place. Measured with a host whose listener starts six seconds late: the router answers 503 until the checks pass at t+9s, and never routes to it before. Also from review: gonka-drain treated "this host is in no backend" as success, which is worst for a host that has just entered DNS and is about to join — it now fails and says so. Duplicate version detection compared with awk's ==, which is numeric when both sides look like numbers, so 1, 01 and 1.0 were one version. The rolling-update suite drained the legacy owner, which is correctly refused. It does not need legacy pinning, so it clears it, and the legacy backend then has nothing routed to it and stays out of the drain.
Readiness for a version was answered from the route table alone, which only says a child process is running. devshardd reports itself unready when its chain subscription drops, and versiond went on answering 200 for that version — the router kept the host in the pool and kept sending it work. ServesVersion now re-asks the child, cached for under a second so a per-second balancer check does not become a per-second probe of the child, and bounded so a child that has stopped answering is not a serving one. gonka-drain loses `out` and `in`. HAProxy identifies a server by its slot in a server-template and slots are reused: a drained host that leaves DNS frees its slot, and the next host to arrive inherits the drain — kept out of rotation with nothing to show why. Admin state belongs to the identity of a process, and the router has no such identity, only an address DNS lent it. Taking a host out of rotation is stopping its versiond, which is graceful by construction and cannot be inherited. `status` stays, read-only. The rolling-update suite pinned traffic by draining the other host; it stops it instead, and the old-generation check now looks only at the host that flipped, since a stopped host has no health to report. Docs caught up with the code. rolling-update.md still specified a transactional router FSM, hostctl checkpoints, membership IDs and schema migrations for tests that no longer exist, and claimed them as the implemented status of Track B. The release guide said the join overlay declares v4 when it declares v4 through v8, and said in one place that a host failing to install a version leaves that version's pool and in another that it stays. The README still demanded the old name grammar that the entrypoint no longer enforces.
The cached probe had the shape the review described. It read the child under the lock, released it, spent up to two seconds on HTTP, then wrote the answer keyed by version name — so a swap in that window let a departed generation decide for its replacement, in either direction, and concurrent callers could land their answers out of order. Nothing invalidated the cache on a route rebuild either, and the test that came with it deleted the entry by hand, so it never exercised any of that. Both that and the fan-out are the same design mistake: answering a question by doing I/O inside the answer. A monitor now runs per generation, refreshing that generation's own flag once a second, and ServesVersion is a pure read. The flag lives on the child, so it is bound to the generation by construction rather than by a check — a late answer can only be written to the child that was asked, and a swap simply ends one monitor and starts the next. A balancer asking every second cannot become a probe every second, and no number of concurrent callers fans out. The flag also carries when it was last refreshed, and an answer older than five seconds is not an answer: a monitor that has stopped must not leave the version frozen at "ready". Tests cover losing and regaining readiness through the real monitor, staleness, and that the current generation alone decides. Docs: the release guide still offered gonka-drain for quiescing and still said a host failing to install a version stays in that version's pool; rolling-update.md still credited gonka-drain with refusing to empty the pool; and the evacuation invariants skipped from 6 to 8 after the earlier renumbering.
Five findings from review, four of them seams where the monitor met the rest of the manager. The route could be seen before the flag. close(c.ready) wakes the swap path, which publishes the new generation itself, and c.serving was stored only after the unlock — so a per-version check landing in that window got 503, and with fall 1 one such answer evicts the host. A fleet-wide rolling update could blink the whole pool. The flag is now seeded before the lock is even taken, so nothing can observe the route without the flag. A crash-restart could leave two monitors on one child. The monitor's context was the generation's, and a monitor caught inside its two-second probe while the process died and came back would see running again and keep writing over the new attempt's answers. The context is now scoped to the process attempt, cancelled and awaited right after proc.Wait() — the probe is context-aware, so the await is prompt. The contract is pinned by a test: after cancel+await, a flipped child provokes no further writes. The empty-VERSIOND_VERSIONS mode never saw live readiness: the coarse /readyz gated on Available, which only says a child process exists. Conditions gains Serving — at least one running child whose vouch is current, computed by the same predicate ServesVersion uses so the two answers cannot drift — and versiondReady requires it. Deliberately "at least one", not "every": requiring every child would let one version's fleet-correlated unreadiness empty the pool and take the healthy versions with it, which is the same trap as gating on Degraded. The reviewer's single-child scenario is covered identically either way; per-version pools remain the precise tool. The legacy /healthz fallback logged a warning per probe, and the monitor probes every second. It logs once per generation now, from the callers that know the generation — which also quiets the startup poller, the other repeat offender. Docs: invariant 9 claimed an unconverged host is not routed to at all; it is not routed to through the host-level pool, while per-version pools serve what it already has, on purpose. The three descriptions of the coarse /readyz now mention the live-readiness requirement.
Serving means "at least one live-ready child", so with VERSIOND_VERSIONS empty a host whose v5 child went unready keeps answering 200 as long as v4 is healthy, and v5 requests keep landing on it hash-dependently — the exact failure the per-version pools exist to prevent, reachable through a supported configuration. Making the coarse answer stricter is not the fix; "every child ready" is the correlated-eviction trap again. The fix is not letting an HA deployment route blind: GONKA_HA with an empty VERSIOND_VERSIONS now fails at startup, with the message naming both remedies. The escape hatch is deliberate and says what it accepts — VERSIOND_ROUTER_ALLOW_COARSE_READINESS=1 — because one legitimate user exists: local-test-net mints version names dynamically inside test scenarios and cannot declare them up front. The join overlay already declares v4 through v8 and never hits the check. test-render pins refusal, override and the non-HA pass-through. Two documentation corrections from the same review. The evacuation doc promised the vouch is withdrawn "within a second"; it normally takes one probe interval, a probe may run up to its 2s timeout, and an unrefreshed answer expires after 5s — it now says that. The README still claimed a host downloading an archive or restarting a child reports 503; after first convergence the latch holds, and a restarting child leaves only its own version's pool.
The review found the override fail-open: any non-empty value, 0 and false included, unlocked coarse readiness. That is an instance of a class, and the class was worse than the instance. Every boolean env here was parsed by each consumer separately, and the two consumers of GONKA_HA disagreed: the router read non-emptiness, so false meant on; devshardd read a known-values switch whose default was off, so a typo silently disabled the storage boot guard — the one value the variable exists to enable. The same deployment setting could be simultaneously on for routing and off for safety. One grammar now, on both sides: 1/true/yes are on, empty/0/false/no are off, anything else refuses to start and names the variable. Each entrypoint parses its booleans once, at the top, into plain variables — GONKA_HA, the coarse override, RENDER_ONLY — so no use site can reinvent the parse; devshardd's HADeployment returns an error for values outside the grammar and the boot guard propagates it instead of guessing "off". Pinned from both directions. Go: every off spelling passes on sqlite, every on spelling refuses it, garbage refuses to boot naming the grammar. test-render: 0/false/NO do not unlock the override, garbage overrides and garbage GONKA_HA are refused with the grammar named, and GONKA_HA=false renders without the Devshard-Ha header — the spelling that used to mean on. Reverting the parser to non-emptiness fails the suite on the first check.
GONKA_HA=' true ' was on for devshardd and a startup failure for the router — the one grammar still had two readings at its edges. Both bool_env helpers now trim leading and trailing whitespace only, matching Go's TrimSpace: deleting all whitespace would accept 't rue', which Go rejects, and recreate the divergence in the opposite direction. The render tests pin ' true ' as on, header stamped, and ' no ' as off, header stripped; both were added before the fix and failed against the untrimmed parser.
Confirmed from review, and worse than stated. edge-api gives its chain probe two seconds, but neither router set `timeout check`, and without it `inter 1s` is HAProxy's entire check budget — connect and read. A chain answering in one to two seconds is slow, not down, yet every check against it fails, and with fall 1 every edge-api leaves the pool at once, since they all share the chain node. The part the review could not see from outside: the probe ran on the request's context, so the checker aborting at 1s cancelled the probe, and the resulting context.Canceled was cached as "chain unreachable" for the next three seconds of checks. The pool did not just blink on a slow moment; the abort kept it down. Both routers now set `timeout check 3s`, above edge-api's 2s probe budget, with the relationship written on both sides of the boundary — the constant in readiness.go names the router setting and the template comment names the constant, since no compiler spans the two. versiond-router gets the same 3s: its /readyz is a memory read, so the tolerance costs nothing, and a host that cannot answer a memory read in three seconds is genuinely wedged. The probe itself now runs on its own context with the readiness budget — the answer is about the chain, not about the caller's patience — and is singleflighted, so an expired cache under concurrent checks costs one chain query instead of one per caller. Tests pin both: eight concurrent checks share one probe, and a hanging probe is cut at its own budget, verdict cached, not at whatever deadline the checker happened to have. Render suites pin the presence of `timeout check` in both routers.
Use the same v5 edge-api and versiond tags in the base and HA Compose models. Add a fast pin contract and a pull-first registry smoke that validates router configs and readiness endpoints on the published artifacts.
Capture the current image before each replacement and restore it when a readiness wait fails before router cutover. Stop failed replicas once HAProxy can exclude them from the live pool. Switch multi-edge traffic to HAProxy after the first ready v5 replica. Exercise rollback, stop, and ordering paths in CI.
Measure the attached v4 cluster and target filesystem before the first PostgreSQL recreate. Reuse the same preflight for detached recovery. Repeat the check in the migration entrypoint so manual Compose runs fail before creating a partial staging copy when free space is insufficient.
Renew the anonymous mount after recovered bind PGDATA has been verified. This prevents Compose from carrying the v4 recovery volume forward. Assert both mount identities in the Docker integration test. Keep the old volume as a rollback copy.
Detect versiond and edge-api topology from existing containers so the standard base-only Quickstart never loads HA services or PostgreSQL. Keep both axes independent, add base versiond readiness, and retain explicit mode flags only for recovery diagnostics.
Remove the first v5 replica from legacy nginx before recreating it, then cut over to HAProxy as soon as that replica passes readiness. Track the active replacement and compensate ordinary failures plus HUP, INT, and TERM so an interrupted wait cannot leave an unready service in rotation.
Mount the persistent target explicitly for both migration source modes so completed staging or published PGDATA bypasses another space check. Cover detached-volume retries with real Docker and allow the recovery runbook to resume through the expected PostgreSQL container.
Probe the v4 /healthz contract repeatedly before accepting an image rollback. Stop a restored service when it does not become stable. Document PostgreSQL's separate preservation contract. Migration keeps both copies but never switches storage history backwards automatically.
Persist the legacy nginx isolation barrier across container restarts and verify rollback with service-specific functional probes. Treat incomplete PostgreSQL staging as reclaimable during preflight and clarify targeted cutover versus full-stack maintenance.
Capture every running and routable version before replacing a versiond service, then require the complete baseline after restoring its image. Cover a single-host rollback that restores v3 but loses v4 and reject it instead of accepting one healthy child.
Use service startup budgets for rollback verification and restore stopped supervisors before capturing a settled availability baseline. Parse health payloads with jq, encode version paths, and verify router rollback against the union of every replica's baseline.
Use capability-aware HAProxy health checks so restored pre-v5 supervisors rejoin only when their real version routes are healthy. Preserve stopped service state, keep rollback armed through route postconditions, and reconcile persisted barriers before router baseline capture.
Bring the upstream bridge 0.2.15 image update into the host evacuation branch without rewriting its existing history.
snevolin
marked this pull request as ready for review
August 4, 2026 21:49
Author
|
Closing this PR so the functionality can be split into smaller, independent PRs that can be reviewed, tested, and merged separately. The work will be resubmitted as focused changes. |
This was referenced Aug 18, 2026
Author
|
This umbrella PR was closed so its functionality could be split into smaller, independently reviewable and mergeable changes. The focused changes extracted from this work are now available as:
All target |
This was referenced Aug 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR targets
upgrade-v0.2.15and implements Track B: gracefulevacuation, replacement, addition, and decommission of a whole
versiondhost, as described in:
rolling-update.mdsection 1.8;It also gives
edge-apithe same readiness-first shutdown contract and ships asafe, topology-aware v4-to-v5 upgrade path for existing join deployments.
The main changes are:
versiondandedge-apinginx pool routers with unprivilegedHAProxy 3.2 images;
SIGTERMonversionda bounded host drain that preserves acceptedrequests and complete SSE streams;
edge-apileave rotation before draining accepted chain queries;devsharddlifecycle behavior as table-driven state machines;
functional rollback, and restart-safe PostgreSQL migration;
Track A, the same-version SHA rolling update of an individual
devsharddgeneration, is described and reviewed in
#1490. Track B builds on those
generation and process-lifecycle primitives at the whole-host level.
Runtime architecture
Each
versiondsupervises onedevsharddchild per approved protocol version;the diagram does not imply a one-to-one
versiond/devsharddrelationship.Why the service routers use HAProxy
Before this change, two nginx OSS containers routed the service pools:
versiond-routerkept/devshard/*sessions sticky by escrow ID and selectedHA or legacy backends by protocol version;
edge-api-routerdistributed stateless/v1/*Tier A reads round-robin.Their startup scripts rendered
VERSIOND_HOSTSandEDGE_API_HOSTSinto staticupstream lists. nginx could refresh the IP of a listed hostname, but it could
not discover a newly added replica or remove a departed one from that list. It
also had no active application-readiness contract, so a listening but starting,
degraded, or draining process could still receive new traffic.
The HAProxy routers implement the pool behavior directly in the data plane:
server-templatefollows all addresses published under the sharedversiond-poolandedge-api-poolDNS aliases;configuration change;
init-state fully-downkeeps a newly discovered address out of rotation untilits first successful check;
hash-key addrmakes the ring independent of DNS answer order and HAProxyslot assignment, so router restarts do not re-home every escrow;
withdrawn, while new work is sent only to healthy replicas.
The HAProxy configuration preserves the relevant nginx behavior: SSE streaming,
forwarding headers, legacy-version pinning, bounded request bodies, and retries
that never replay non-idempotent application work.
Routing and readiness
versiond router
Pool membership and route eligibility are separate questions:
VERSIOND_POOL_HOST.continue serving every other healthy route.
The versiond health check is intentionally two-stage so v5 HAProxy can safely
run during a mixed-version cutover or a rollback to v4:
A v5
503is authoritative and removes a starting or draining host. Acceptingthe pre-v5
404never admits a process by itself: the real host or version routemust still return
200.VERSIOND_VERSIONSdeclares the routable version window. Once that list isnon-empty, an undeclared version fails closed with
503instead of fallingthrough to a host by accident. Versions in
VERSIOND_NON_HA_VERSIONSuse thesame per-version check but remain pinned to
VERSIOND_LEGACY_HOST, because theirSQLite sessions cannot fail over.
The request hash is derived inside HAProxy. Session paths use the escrow ID, so
every request for the same live session reaches the same
versiond; non-sessionpaths hash by normalized path. Client headers cannot choose the backend.
versiond endpoints
All readiness endpoints are on the traffic listener:
GET /healthzGET /readyzGET /readyz?version=<v><v>Child readiness is monitored continuously. A running child that loses its chain
subscription loses its readiness vouch and leaves the corresponding pool. The
host-level
Convergedcondition latches after the full desired set has run once,so an ordinary same-name child replacement does not evict every host at once.
Reconcile errors remain observable without hiding routes that are still serving.
edge-api router
edge-api-routerdiscoversEDGE_API_POOL_HOST, balances healthy replicasround-robin, and checks
GET /readyz.edge-apireadiness is a bounded, cachedchain probe, so an instance that cannot reach the chain leaves rotation and
rejoins automatically when the dependency recovers.
HA storage guard
The HA overlay sets
GONKA_HA=true, which makesdevsharddrefuse to startunless its storage is fail-closed Postgres. HAProxy also removes any
client-supplied
Devshard-Haheader and stamps its own trusted value:number of usable servers in the backend selected for the request;
devsharddvalidates the header again against its active storage. This catchespartial rollouts and accidental multi-host scaling even when the deployment flag
was omitted.
Graceful shutdown
versiond host evacuation
The operator uses the ordinary container lifecycle. For example, from
deploy/join:The safe ordering is owned by
versiond:VERSIOND_DRAIN_ANNOUNCEdefaults to five seconds. During that window the hostalready fails readiness but still accepts work, giving HAProxy enough time to
withdraw it before admission closes. A host that never reached
servingskipsthe announcement and drains directly because it was never eligible for traffic.
Every proxied request acquires a host admission lease before forwarding. The
lease is released only when the handler finishes; for SSE that means the full
lifetime of the stream. A child generation has its own target lease, so a
rolling replacement cannot reap the exact process still serving that response.
New requests go to a surviving host. HA sessions can recover there from shared
Postgres. A legacy SQLite owner cannot be evacuated while
VERSIOND_NON_HA_VERSIONSis non-empty.Temporary stop, replacement, addition, and permanent decommission are described
in the host evacuation runbook.
Permanent membership is represented by the persisted Compose replica count,
for example
VERSIOND2_REPLICAS=0, so a later Docker daemon restart cannotsilently recreate a decommissioned service.
edge-api shutdown
edge-apifollows the same readiness-first sequence:The default five-second
EDGE_API_DRAIN_ANNOUNCElets HAProxy remove thereplica before Echo closes the listener. Accepted queries then receive up to
EDGE_API_SHUTDOWN_BUDGETto complete. A repeated signal shortens announcementor forces remaining connections, and Docker's stop grace remains the final
backstop.
Budgets and escalation
versiondfixes one absolute deadline when shutdown begins. Announcement, pollunwind, host leases, child drain, child process grace, and HTTP shutdown all
spend from that same budget; sequential phases cannot add independent maximums.
VERSIOND_DRAIN_ANNOUNCE5sVERSIOND_HOST_SHUTDOWN_BUDGET25mVERSIOND_STOP_GRACE_PERIOD30mSIGKILLreserveEDGE_API_DRAIN_ANNOUNCE5sEDGE_API_SHUTDOWN_BUDGET2mEDGE_API_STOP_GRACE_PERIOD3mSIGKILLreserveRepeated
SIGTERMis idempotent;SIGINTexplicitly forces versiond teardown.Budget expiry forces remaining HTTP work and children, then confirms process
reap before exit. The poll worker receives at most 10% of the remaining host
budget, capped at five seconds. If it ignores cancellation, the manager drain
barrier already prevents new generations and restarts, so shutdown continues.
State machines
1. versiond host lifecycle
starting: not ready and not accepting;serving: ready, accepting, and reconciling children;announcing: unready but still accepting while HAProxy removes the host;draining: admission and desired-state changes are closed;stopping: child and HTTP graceful shutdown is running;forcing: deadline or explicit escalation is closing remaining work;stopped: HTTP has stopped and every child has been reaped.The transition table owns legality, admission policy, and readiness policy.
starting -> announcingis forbidden because a host that never served must notopen admission while shutting down.
2. Child generation lifecycle
The generation FSM owns route publication, exact-target retirement, drain, and
restart.
retiringremoves that generation from new routing before lifecycledrain starts. The manager rechecks the generation under its mutex immediately
before every process fork, so host drain and generation retirement form a real
start barrier rather than a timing assumption.
3. Supervised OS process lifecycle
This FSM owns process-group
SIGTERM, escalation toSIGKILL, and reap. Onlythe process-exited event produced by
cmd.Waitreachesexitedand closescompletion.
4. devshardd request lifecycle
Readiness, admission, drain state, and the in-flight counter come from one table
under one mutex.
drainingis terminal and rejects new non-lifecycle work.Safe v4-to-v5 deployment transition
The steady-state readiness and drain contracts protect v5 operation, but an
existing v4 installation cannot safely reach that state through a blind
full-model Compose recreate. The deployment contracts are materially different:
v4 uses static nginx upstreams and has no
/readyz; v5 uses DNS-discoveredHAProxy pools and active readiness. The HA PostgreSQL mount also changes from an
anonymous image volume to persistent bind-mounted
PGDATA.deploy/join/upgrade-devshard-v5.shprovides the compatibility bridge. Theoperator runs one command from
deploy/join:./upgrade-devshard-v5.shThe driver sources
config.env, checks its dependencies, and detects twoindependent topology axes from the existing containers:
versiondversiond2, andversiond-routeredge-apiedge-api2,edge-api3, andedge-api-routerThe standard Quickstart therefore remains a single-service upgrade and does not
create HA-only services. HA overlays are loaded only for installations that
already use them. Every replacement is targeted with
--no-deps; unrelatednode, chain, proxy, bridge, Tmkms, explorer, and ML services remain running.
Ordered cutover
The temporary nginx barrier removes the active replacement from the rendered
upstream and reloads nginx without terminating established connections. An
after-render hook persists that isolation across a router container restart.
Once the first v5 replica is healthy, HAProxy's active checks protect later
replacements.
Before cutover-sensitive replacements the script captures the immutable source
image ID, whether the service was running, and every version route that was
available.
EXIT,INT,TERM, andHUPuse the same compensation path. Afailed replacement before the readiness-aware router is installed restores the
captured image and original running/stopped state, or stops a newly introduced
service, then requires three consecutive functional probes. After HAProxy owns
an edge-api pool, an unready replacement is stopped while the surviving replicas
continue serving. Versiond rollback must restore every baseline version;
edge-api rollback must execute the chain-backed
/v1/versionsquery; routerrollback must route the union of all replica baselines.
The mixed-version HAProxy check described above lets a restored v4 versiond
rejoin only when its real routes work. A v5 process that explicitly returns
503remains out of rotation. Temporary rollback tags are retained after afailed operation and removed only after the complete upgrade succeeds.
PostgreSQL migration and recovery
The v4 HA overlay inherits the Postgres image's anonymous
/var/lib/postgresql/datavolume. v5 storesPGDATAunder the stableDEVSHARD_POSTGRES_DATA_DIRbind mount.Before recreating PostgreSQL, the upgrade driver mounts the source read-only,
measures it with
du, checks the target filesystem withdf, and requires thesource size plus a 10% reserve. The entrypoint copies into a staging directory,
validates
PG_VERSION, syncs the copy, and atomically publishes it. The sourcevolume remains unchanged as a physical recovery copy.
Startup fails closed when versiond artifacts imply an existing installation but
the source database is missing; it never silently initializes an empty HA
database. A recovery overlay can reattach an explicitly selected detached v4
volume and reuse the same restart-safe preflight and atomic migration. Partial
staging is reclaimable, completed staging is reusable, and the source is
detached only after bind-mounted
PGDATAis verified.The exact procedure and recovery boundaries are documented in the
0.2.15-v5release guide.Failure policy
503Test coverage
The full-stack
TestVersiondHostEvacuationuses two real
versiondcontainers, HAProxy, shared Postgres, and a pausedOpenAI-compatible SSE response. It verifies:
SIGKILLbackstop;Focused coverage includes:
reap under
go test -race;undeclared-version refusal;
version-baseline tests;
restartability tests;
images.
Relevant commands:
Dedicated workflows run the host-evacuation acceptance test, real PostgreSQL
upgrade/recovery tests, and release-image smoke checks for the files that affect
those contracts.
Deployment and documentation
Important deployment settings:
VERSIOND_POOL_HOSTandEDGE_API_POOL_HOSTresolve to every replica in theapplicable pool;
VERSIOND_VERSIONSdeclares every version the HA router may serve;VERSIOND_LEGACY_HOSTis required whileVERSIOND_NON_HA_VERSIONSis non-empty;VERSIOND_STOP_GRACE_PERIODexceedsVERSIOND_HOST_SHUTDOWN_BUDGET;EDGE_API_STOP_GRACE_PERIODexceeds the edge-api announce and shutdownbudgets;
0.2.15-devshard-v5image tag.Supersedes #1489. This branch applies Track B to
upgrade-v0.2.15and includesthe release transition needed to reach the new DNS-discovered, readiness-gated
runtime safely from existing v4 join deployments.