Goldseam

Named for kintsugi — the Japanese art of repairing a break with gold so the seam becomes the most beautiful part of the object. Goldseam is that seam: it deliberately kills a llama.cpp process and rejoins it warm via a KV snapshot→restore, so the break is the mechanism, not damage.

Warning

Maturity / AI authorship. This codebase sits somewhere between "copilot" and "auto" on the ai-declaration.md scale — it was extruded from a spec authored by Claude under human guidance, then iteratively adjusted and debugged against a single box. The warm-swap mechanism is validated end-to-end on that one hardware configuration and in the unit + example-policy smoke test suite, but it is an early, specialized tool. It won't destroy your system, but it may also not behave as expected. Don't run it blind — be ready to read the code and fix what you hit. See KNOWN_ISSUES.md for the current limitations and the gotchas that were learned the hard way.

Goldseam is a Rust reverse proxy that owns your llama.cpp model processes and keeps conversations warm across GPU swaps by moving their KV/recurrent state instead of re-prefilling. Swap a model out of the GPU and back in, and the conversation resumes where it left off — reprocessing only the new tokens, not the whole context.

What runs where — which model loads, when, on which GPUs — is decided by a small TypeScript policy you can edit and hot-reload, not by hardcoded rules. It works on any GPU layout via a per-GPU VRAM budget: there's no "both cards" concept, coexistence just falls out of each model's declared footprint.

client ──▶ :8080 (OpenAI- & Anthropic-compatible) ──▶ proxy ──▶ llama.cpp child
             route by request `model`; if it isn't resident, hold the request,
             swap it in (warm if possible), then forward — never a 503.

Why it's interesting

• Warm KV-transfer swaps. Snapshot a conversation's KV state to a RAM blob, kill the process to free the GPU, later respawn and restore — the conversation resumes warm. In testing, a ~4,800-token context came back after reprocessing 19 tokens instead of all 4,800.
• Demote-to-coexist. The KV blob is layout-portable, so a live dual-GPU chat can be demoted onto a single card to make room for a second model — without losing its warmth.
• Policy is code, not config. Scheduling is a sandboxed TypeScript decide(state, req) function (runs in QuickJS, type-checked by tsgo, hot-reloaded on save). A bad policy can't wedge the box: the engine validates every plan against the real VRAM budget and falls back to a safe cold load.
• Any GPU layout. One card, two, eight, mixed sizes — described entirely by per-GPU VRAM footprints in catalog.toml. No model-class tags, no groups.

How it compares to llama-swap and ananke

llama-swap and ananke both solve a similar problem — on-demand model swapping behind one OpenAI-compatible endpoint. llama-swap is the mature, backend-agnostic swapper; ananke adds GPU/VRAM-aware placement and a management surface. Goldseam is narrower and more opinionated: it trades both tools' backend-generality for warm, KV-preserving swaps and scheduling-as-code, on llama.cpp specifically.

	Goldseam	llama-swap	ananke
Model switch	Warm — KV snapshot/restore, resumes mid-conversation	Cold — process teardown + full re-prefill	Cold — drain + restart
Backends	llama.cpp only (needs a patched build)	Backend-agnostic — any OpenAI-compatible server / command	llama-cpp template + command template (any binary/container, e.g. ComfyUI, vLLM)
VRAM tracking	Per-GPU VRAM budget; coexistence emergent from declared footprints	None — you ensure fits manually	Automatic, GGUF-aware estimation from the model file
Scheduling	TypeScript decide() policy: sandboxed, hot-reloaded	YAML config: groups, TTL unload, profiles	TOML [[service]] blocks: lifecycle, priority, idle timeout
Multi-GPU placement	Per-GPU VRAM budget; emergent coexistence, demote-to-coexist, load-driven autoscaling	Manual, via groups	Packer picks GPUs by free VRAM; layer/row/tensor split modes; MoE expert offload to CPU
Eviction	Policy-driven (demote/coexist before evict; cold fallback last)	Solver-based matrix of valid concurrent sets	Priority-based; higher priority displaces dormant on-demand
Management surface	/status JSON + /routine/{name}	REST API + web UI	REST + WebSocket events + anankectl CLI (chat TUI, logs, oneshot jobs)
Hot-reload	policy.ts mtime-poll with tsgo gate	Config reload	Config reload with GGUF preflight
Setup	Patched llama.cpp baked into a Docker image	Single static binary, no patches	Stock llama-server on PATH, NVML; no patches
Maturity	Early, specialized use	Mature, widely used	Early (per README)

Reach for llama-swap when you want a simple, proven swapper, you run non-llama.cpp backends (vLLM and friends), or a cold re-prefill on each model switch is acceptable.

Reach for ananke when you want GPU/VRAM-aware placement without hand-authoring a per-layout footprint table — its GGUF estimator sizes each model for you — and you'd like a first-class management surface (CLI, WebSocket events, oneshot jobs) plus the ability to front non-llama.cpp tools (ComfyUI, vLLM) alongside your GGUF models. Cold swaps only, and the README warns it's not yet mature.

Reach for Goldseam when you switch models or layouts often enough that cold re-prefill hurts, you're on llama.cpp and can run the patched build, and you want a multi-GPU box to keep conversations warm across swaps — with placement decisions expressed as code.

They can also compose (advanced). You can run Goldseam as a single backend behind llama-swap or ananke — register one Goldseam upstream with every Goldseam model name as an alias, so the model field passes through and Goldseam still picks the layout and does the warm swap. This earns its keep when the outer proxy is arbitrating the same GPUs across backends Goldseam doesn't handle (e.g. an image-gen tool): it frees the card when you need it elsewhere and brings Goldseam back afterward. Two things to get right — give Goldseam dedicated GPUs so the two layers aren't both managing the same VRAM, and note that an outer-proxy-triggered restart drops Goldseam's in-VRAM warm state (conversations cold-prefill on the next request; reattaching persisted blobs on restart isn't wired up yet). If your other models don't share Goldseam's GPUs, skip the nesting and just run the two side by side. Ananke is the more natural outer layer here — its GGUF estimator and packer already understand per-GPU free VRAM, so giving Goldseam a dedicated GPU with a placement_override matches how ananke thinks about resources.

Quick start

cp catalog.example.toml catalog.toml    # edit for your GPUs + GGUF paths
cp policy.example.ts     policy.ts       # the default policy — tweak later
# point docker-compose.yml's volumes at your model + config dirs, then:
docker compose up -d
curl -s localhost:8080/status | jq

On startup the proxy cold-loads the default model and runs a warm-restore self-test before accepting traffic — if the patched llama.cpp is missing or broken, it refuses to serve rather than silently re-prefilling. Full configuration and policy-authoring guides are in docs/.

Build for your GPU

There's no prebuilt image — the patched llama.cpp is compiled from source when you build, so the build targets a specific CUDA architecture. The default is 86 (Ampere consumer — RTX 3090/3060). Set it for your card: find its compute capability in NVIDIA's GPU table and drop the dot — 8.9 → 89, 9.0 → 90, 12.0 → 120 (a list works too, e.g. "80;86;89"):

CUDA_ARCHITECTURES="89" docker compose build      # or put CUDA_ARCHITECTURES in a .env file

The default 86 also runs binary-compatibly on Ada (8.9), but for native, well-tuned kernels set the arch that matches your card — A100 (8.0), Hopper, Blackwell, and pre-Ampere GPUs all need their own value. Power users can pass extra llama.cpp cmake flags via the LLAMA_EXTRA_CMAKE_FLAGS build arg.

Requirements

• NVIDIA GPU(s) and the NVIDIA Container Toolkit — follow NVIDIA's install guide, then sudo nvidia-ctk runtime configure --runtime=docker.
• Docker. Set shm_size: 256mb if you use multi-GPU tensor-split layouts (NCCL needs the shared memory; Docker's 64 MiB default hangs on spawn). Not needed for single-GPU setups.
• GGUF model files on disk, referenced from catalog.toml.
• A RAM-backed slot_dir (tmpfs) is strongly recommended for fast KV snapshot/restore. NVMe works too; the proxy only warns if it's on slower storage. Blobs run ~56 KB/token for hybrid/recurrent models, so size your tmpfs for however many conversations you keep warm.
• A patched llama.cpp — baked into the Docker image at build time (three patches: warm restore, divergent-prompt safety, and persistent prompt-cache pool across swaps). Nothing to apply by hand; see docs/patches.md.

Endpoints

All llama-server endpoints are routed through the proxy. They fall into three groups:

Model-routed (body has model field; non-resident model is held and swapped in, not rejected):

• POST /v1/chat/completions, /chat/completions — OpenAI chat completions
• POST /v1/completions — OpenAI completions
• POST /v1/embeddings — OpenAI embeddings
• POST /v1/responses, /responses — OpenAI Responses API
• POST /v1/messages — Anthropic Messages API (Claude Code via ANTHROPIC_BASE_URL)
• POST /v1/messages/count_tokens — Anthropic token counting
• POST /v1/rerank, /v1/reranking, /rerank, /reranking — reranking

Active-model pass-through (forwarded to the currently-routed model; 503 if nothing resident — these operate on the live child, not a requested model):

• GET /props (also POST), GET /slots, GET|POST /slots/:id — live child state
• GET /v1/health, GET /metrics — child health / Prometheus metrics
• GET|POST /lora-adapters — LoRA adapter management
• POST /v1/chat/completions/control — control an in-flight completion
• POST /completion, /completions, /embedding, /embeddings — legacy inference
• POST /infill — code infilling
• POST /tokenize, /detokenize, /apply-template — tokenizer / template
• POST /v1/audio/transcriptions, /audio/transcriptions — audio transcription

Catalog-answered (answered by the proxy from catalog.toml, not forwarded):

• GET /v1/models, GET /models — all canonical names + aliases

Control plane (proxy's own):

• GET /health — liveness (ok)
• GET /status — model states, layouts, ports, per-GPU VRAM, last_policy_error
• POST /routine/{name} — run a named policy routine (e.g. researchMode)

Query strings are preserved on all forwarded endpoints (so /slots/:id?action=save and similar work). All POST routes forward as streaming when the client asks for stream:true. Details in docs/.

Build & test

cargo fmt --check
cargo clippy --all-targets -- -D warnings
cargo test                    # no GPU needed: VRAM math, plan validator, policy smoke tests
python3 tests/roundtrip.py    # end-to-end warm-swap proof (needs a running proxy + GPU)

Documentation

Configuration, policy authoring, the llama.cpp patches, and troubleshooting all live in docs/. See DESIGN.md for how it works under the hood, and KNOWN_ISSUES.md for current limitations.

License

MIT — see LICENSE.txt.