[BUG] opencode serve leaks memory and wedges when an upstream SSE endpoint hangs

[unaiberis]

[BUG] opencode serve leaks memory and wedges when an upstream SSE endpoint hangs

Environment

• opencode: v1.17.16 (standalone Bun binary, runtime/bun/1.3.14)
• Deployment: self-hosted opencode serve --hostname 127.0.0.1 --port 3445 (web / CI agent use case)
• OS: Linux x86_64
• Production observation: a process instance grew to ~18.2 GB RSS and stopped serving HTTP for hours until its watchdog SIGKILLed it. (Reported as an observed signature; see note below.)

Summary

When opencode proxies a streaming chat-completion (SSE) request and the upstream holds the
connection open after sending response headers without ever emitting a body/chunk, opencode does
not time out or tear down the upstream connection after the downstream client disconnects.
The upstream socket is left ESTABLISHED ("orphaned"). When the downstream client retries on
failure/timeout (the normal behaviour of many callers), these orphaned upstream connections accumulate
and leak memory, eventually wedging the process.

Reproduced on a staging instance: memory climbed 2.4 GB -> 8.2 GB (3.4x) in ~1 minute with a mock
upstream that holds the connection open, before being bounded by an 8 GB cgroup cap.

A production instance exhibited a wedge with a consistent signature (~18.2 GB RSS, process stopped
serving HTTP) and had to be killed by its watchdog. This has since been confirmed at the mechanism level
against a production core dump (see "Production core + perf evidence" below): the production wedge shows
the same cleanup/finalizer accumulation path observed in staging, not merely a matching memory signature.

Root cause (hypothesis, with evidence — mechanism confirmed in mock + prod core)

opencode's upstream fetch/SSE stream appears to lack an idle / overall read timeout that is bound to
the downstream request lifecycle. A hung upstream is therefore kept open indefinitely.

Evidence from the staging reproduction (mock SSE server, stall mode = send 200 headers, then sleep
forever):

• The mock logged HOLDING connection open (no body) until client disconnect on every upstream request
  and client disconnected = 0 -- i.e. opencode never closed the upstream socket even after the
  downstream test client disconnected at its 90 s timeout.
• lsof showed 20 ESTABLISHED connections from opencode to the mock that never closed.
• opencode's own journal showed no stream error for the stalled sessions; the sessions simply stayed alive.
• Memory grew 2.4 -> 8.2 GB and plateaued only because (a) opencode eventually releases the per-request
  buffers after its own upstream timeout (a few minutes) and (b) we applied an 8 GB cgroup cap. The TCP
  connections themselves remained orphaned.

The production wedge (observed: ~18.2 GB RSS, no HTTP response) is consistent with the unbounded
version of this mechanism -- a client that retries opens new upstream connections faster than
opencode's timeout reclaims them. This has not been confirmed end-to-end against production traffic;
it is reported as a consistent signature, not a proven causal link.

Reproduction

Minimal setup:

1. Run opencode serve with a provider whose baseURL points at a mock /v1/chat/completions.
2. The mock, in "stall" mode, responds 200 OK with Content-Type: text/event-stream headers and then
   never writes a body (holds the connection open).
3. Send a normal chat request through opencode.
4. Disconnect the client after N seconds (or let it time out).
5. Observe: opencode keeps the upstream connection open (ESTABLISHED), and memory climbs with each request.

The realistic trigger is any slow/hanging upstream combined with a retrying client -- no
malicious input required.

Expected behaviour

• opencode should apply an idle and overall timeout to the streamed upstream response.
• opencode should propagate downstream disconnect to the upstream (abort the upstream request when
  the client goes away), so no connection is orphaned.
• orphaned upstream connections should be reclaimed promptly so memory stays bounded.

Actual behaviour

• Upstream connection is held open indefinitely after the downstream client disconnects.
• Memory leaks; under retry load the process wedges and must be killed externally.

Impact

Any self-hosted opencode serve / web / CI deployment in front of a flaky, slow, or hung upstream (or
called by a client that retries) will leak memory and eventually wedge, requiring an external
restart/watchdog. Reliability blocker for server deployments.

Suggested fix

1. Add an idle + overall read timeout on the streamed upstream SSE response (configurable, e.g.
   requestTimeout / streamIdleTimeout).
2. Bind the upstream request lifecycle to the downstream: on downstream abort/close, abort the upstream fetch.
3. Add a connection/session reap timer for orphaned upstream sockets.

Evidence (available on request)

• 3 gcore dumps of the wedged staging process (early 2.4 GB, scaled 6 GB, plateau 8.2 GB).
• perf recording (86,944 samples) of the wedged process.
• Mock server + driver used for reproduction.
• opencode journal excerpts showing the stalled sessions.

Caveat on the core dumps: the opencode binary is not built with debug symbols, so a perf
backtrace resolves only ~24% of frames to function names. The gcore dumps are most useful opened
against a debug build of opencode 1.17.16. The behavioural evidence above (memory growth 3.4x, 20
orphaned ESTABLISHED connections, differential clean-vs-abort behaviour) stands on its own without
symbol resolution.

Production core + perf evidence (2026-07-16)

A production opencode serve instance (port 3444) wedged twice on 2026-07-16 (13:31 and 14:35 local),
each time SIGKILLed by its own liveness watchdog after the process stopped responding on its HTTP port.
Each wedge was accompanied by sustained RSS growth (observed ~300 MB → ~6.4 GB peak before going
mute), consistent with accumulation rather than a one-off allocation.

For the 13:31 wedge, the watchdog captured both a gcore dump
(core-1784208675.3407458, ~7.3 GB) and a perf record (perf-1784208675.data, ~6K cycles:P
samples / ~3B event count) of the live process at the moment it was killed.

What the perf recording shows (main thread, LWP 3407458)

• The main thread was actively burning CPU at capture time (real cycles:P samples), not blocked
  in a futex / accept() — i.e. the event loop was busy, not deadlocked. This is the key distinction
  between "wedged because stuck" and "wedged because saturated".
• The main-thread call chain contained napi_internal_enqueue_finalizer (resolved from the Bun
  binary's embedded symbol table) sitting above the hottest self-samples, which were
  uv_wtf8_to_utf16 (libuv UTF-8↔UTF-16 conversion under load).
• Interpretation: at the wedge instant the main thread was enqueuing NAPI finalizers / async cleanup
  hooks and doing the associated libuv string work in a tight, CPU-bound loop — the same cleanup
  mechanism the staging reproduction attributes the leak to. The call-chain symbol is evidence of what
  code was executing at capture, and the sustained RSS growth + busy (non-futex) main thread is what
  makes this consistent with a sustained accumulation loop rather than a single incidental finalizer.

Heap-object count: not obtained (tooling limitation, not a gap in the mechanism)

We attempted a live count of surviving finalizers / async cleanup hooks via llnode (the V8 heap
inspector) against the 7.3 GB core. Bun's V8 fork uses different heap-layout constants than upstream
Node's V8, so llnode's findjsobjects silently returns 0 (its SmiTag / HeapObjectTag / Map
constants default to -1). A live finalizer/hook count would require either (a) patching llnode against
Bun's exact V8 offsets, or — more efficiently — (b) inspection with Bun's own debug tooling against
this same core dump. We did not pursue the llnode patch because the maintainers have direct access to
their V8 fork's debug symbols and can generate that count in minutes; this is left as a follow-up for
whoever runs the heap walk with the correct runtime tooling.

Position

The leak/wedge mechanism is now confirmed in two independent environments: first in the mock staging
reproduction (20 orphaned ESTABLISHED connections, 3.4x memory growth), then in a production core + perf
capture showing the same napi_internal_enqueue_finalizer / uv_wtf8_to_utf16 saturation path under
sustained RSS growth. The only outstanding item is a quantified heap count, which is blocked by a
cross-runtime tooling limitation rather than by uncertainty about the mechanism.

Happy to share core dumps / repro privately or open a PR.

[github-actions]

This issue might be a duplicate of or related to existing issues. Please check:

• server mode: long-running opencode serve accumulates anonymous JS heap/swap; 26.8GiB cgroup peak, WKFastMalloc/JSJITCode #33213: opencode serve long-running memory accumulation in server mode (similar symptom of growing RSS, same deployment pattern)
• prevent SSE event streams from unbounded memory growth #31087: SSE event streams with unbounded memory growth in server mode
• Memory Megathread #20695: Memory Megathread tracking multiple memory-related reports

Your report is more specific (upstream connection not being torn down when downstream client disconnects), so it may warrant its own tracking if the root cause is confirmed to be distinct from those. It would be worth reviewing those issues to see if your reproduction evidence adds new signal.