Streaming BlobAdapter methods: the byte-array contract forces full buffering of every attachment (deferred until triggers hit)

[cuibonobo]

Deferred design note from the server design review (haverstack/server#46 is the bounded-buffer symptom fix; this is the contract-level cause). Filed to be on record with explicit wake-up triggers — not proposed for the current implementation phase.

Problem

The blob contract speaks whole byte arrays:

putAttachment(data: Uint8Array): Promise<FileId>;
getAttachment(fileId: FileId): Promise<Uint8Array>;

The wire is not the problem — POST /attachments takes a raw binary body and downloads return one; both are streaming-shaped at the HTTP level. It's this contract in the middle that forces every layer above it to materialize the full payload: the server buffers uploads to call putAttachment and buffers downloads because it receives a complete Uint8Array (whose byteLength is also where Content-Length and the metadata record's size come from). Same shape propagates through Stack/ScopedStack.putAttachment and adapter-api.

Consequences at scale: a served stack's memory floor is roughly concurrent transfers × payload size × ~2. Tolerable at the 50 MB default ceiling; not tolerable the moment the ceiling rises for real media — and self-hosting on small boxes is this project's deployment story. Whole-buffer download also forecloses HTTP Range (Accept-Ranges/206), which is what seekable video/audio playback needs.

Sketch (additive, capability-flagged)

putAttachmentStream(data: ReadableStream<Uint8Array>): Promise<{ fileId: FileId; size: number }>;
getAttachmentStream(fileId: FileId, opts?: { offset?: number; length?: number }): Promise<{ stream: ReadableStream<Uint8Array>; size: number }>;

• Web Streams as the primitive (runtime-neutral; Node fs converts via Readable.toWeb). Core ships buffering shims both directions so existing adapters keep working — additive in place per the standing policy (_grant records store the grantee in entityId, the authorship field — move it into GrantContent #57), flagged like Orphaned blob GC: deleting the last referencing record leaves bytes on disk forever #64's listFiles().
• putAttachmentStream returns size because the caller no longer holds the bytes and the _attachment@1 record needs it.
• Content addressing means the fileId is unknowable until the stream ends: disk adapter writes a temp file while hashing incrementally, then renames to the hash-named path (same atomic-rename idiom as Single-writer storage model: durability fixes for adapter-local + spec/README guidance on multi-app topology #45); dedup is discovered at stream end (hash exists → discard temp). No worse than today's full-buffer hash.
• Web Crypto has no incremental hashing (crypto.subtle.digest wants the whole buffer). Node's createHash streams fine, so blob-adapter-disk is unaffected — but a future browser/OPFS blob adapter needs a small JS/WASM SHA-256 or falls back to the buffering shim. Settle this when that adapter gets scheduled.
• No bad interactions with the review's attachment work: Served Content-Type is nondeterministic when multiple _attachment records share a fileId — make mimeType a property of the bytes #65/Dangerous-type forcing must cover all three Content-Type sources on attachment download #66 are metadata-level, Orphaned blob GC: deleting the last referencing record leaves bytes on disk forever #64 walks records + listFiles(), ScopedStack's access predicate runs before bytes flow. Wire needs zero changes for streaming itself; Range support is additive HTTP (future PATCH contract mismatch: spec defines content-only merge patch, APIAdapter sends record-field envelopes #52 fixtures if/when specced).

Why deferred, and what wakes it

Unlike the wire-contract issues, waiting here bakes in no breakage — the change stays additive whenever wanted, and the wire is already the right shape. Wake-up triggers, any one sufficient:

1. Raising the attachment ceiling past ~100 MB (or a real media use case: video playback wanting Range)
2. Memory-constrained self-hosting reports
3. A browser/OPFS blob adapter getting scheduled (forces the incremental-hash decision anyway)

One cheap thing worth doing now: haverstack/server#45's upload-gating fix will likely add a scoped bytes-entry point to core — give that new signature room to accept Uint8Array | ReadableStream<Uint8Array> later without another round of churn.

Work items (when woken)

• Contract methods + capability flag + buffering shims in core
• blob-adapter-disk: temp-file + incremental-hash + rename implementation; offset/length reads
• Server: stream request body → adapter; stream responses; Content-Length from metadata/stat; Range support as follow-up
• Spec: §Adapters (streaming capability), §Attachments (Range, if adopted); PATCH contract mismatch: spec defines content-only merge patch, APIAdapter sends record-field envelopes #52 fixtures for whatever lands

Refs #45 (atomic-rename idiom), #46 (adapter packaging), #64 (capability-flagged adapter addition precedent), #56 (maxAttachmentBytes discovery — enforceable mid-stream once this lands), #57 (additive-in-place policy), haverstack/server#46 (symptom fix), haverstack/server#45 (signature coordination).

[cuibonobo]

Cross-ref from the 2026-07 design-assessment pass (docs/design-assessment-2026-07.md): still correctly deferred — none of the wake-up triggers here have fired. But two assessment issues rewrite the blob write path: #106 (§A1 — folding bytes+metadata into one atomic putAttachment) and #121 (§F8 — temp-file-and-rename on DiskBlobAdapter.putAttachment). Per this issue's own "one cheap thing worth doing now" note, whoever implements those should leave the upload signature room to accept Uint8Array | ReadableStream later, so streaming stays additive and isn't foreclosed. No action on #72 itself.

---

Generated by Claude Code