A benchmark measuring each phase of LLM instance cold start separately, from disk to first served token.
Closes the loop with the autoscaling simulation:
slo-aware-autoscaling-sim used startup_delay as a parameter (5s, 15s, 30s, 60s) model-cold-start-bench measures where those numbers actually come from
For full methodology and design decisions see design.md.
Phase 1: weight loading disk -> CPU RAM Phase 2: weight transfer CPU RAM -> GPU VRAM (PCIe) Phase 3: CUDA initialization first kernel launch overhead Phase 4: KV cache allocation pre-allocating serving memory Phase 5: warmup forward first forward pass, JIT compilation
Hardware: NVIDIA GeForce RTX 2070 Measured PCIe bandwidth: ~3.1 GB/s effective
Phase breakdown:
GPT-2 (124M): weight loading: 430ms (78%) weight transfer: 76ms (14%) CUDA init: 6ms (1%) KV cache alloc: 1ms (0%) first forward: 38ms (7%) cold start total: 551ms warmed TTFT: 8ms multiplier: 68x
GPT-2 Medium (355M): weight loading: 598ms (74%) weight transfer: 195ms (24%) CUDA init: 1ms (0%) KV cache alloc: 4ms (0%) first forward: 15ms (2%) cold start total: 813ms warmed TTFT: 13ms multiplier: 64x
GPT-2 Large (774M): weight loading: 815ms (61%) weight transfer: 487ms (36%) CUDA init: 1ms (0%) KV cache alloc: 10ms (1%) first forward: 23ms (2%) cold start total: 1337ms warmed TTFT: 22ms multiplier: 62x
Pre-warming savings:
Prewarm weights on GPU: 92-97% cold start reduction Prewarm weights + CUDA + KV: 93-98% cold start reduction
Extrapolated cold start (no warm pool):
7B (14GB FP16): ~20s 13B (26GB FP16): ~37s 70B (140GB FP16): ~200s
Weight loading from disk dominates cold start. 78-74% of cold start is disk read for smaller models. Grows to 61% for 774M as PCIe transfer becomes more significant.
PCIe transfer is the fastest-growing phase with model size. At 774M: 36% of cold start. At 7B+ it becomes a major fraction.
CUDA init, KV allocation, and JIT compilation are negligible. Under 10% combined in all tested models.
Pre-warming weights on GPU eliminates 92-98% of cold start. From 551-1337ms down to 15-45ms.
The autoscaling startup delays are validated by this benchmark. startup_delay=5s -> weights pre-loaded on GPU (warm pool) startup_delay=15s -> weights in CPU RAM startup_delay=30s -> 7B model loading from disk startup_delay=60s -> 13B model loading from disk
Optimizing startup is more valuable than optimizing autoscaling logic. The autoscaling simulation showed startup delay dominated SLO outcomes. This benchmark shows warm pools reduce it from 20-40s to milliseconds.
model-cold-start-bench/ ├── src/ │ ├── init.py │ ├── config.py │ ├── phases.py │ ├── benchmark.py │ └── analysis.py ├── results/ ├── plots/ ├── LICENSE ├── design.md ├── README.md ├── requirements.txt └── run.py
python3 -m venv venv source venv/bin/activate pip install -r requirements.txt python run.py
Outputs:
results/results.csv results/summary.txt plots/phase_breakdown.png plots/startup_timeline.png plots/prewarm_savings.png plots/scaling_with_model_size.png
This project closes the startup delay loop:
slo-aware-autoscaling-sim: quantified how startup delay dominates SLO compliance model-cold-start-bench: measures the source of that delay and how to reduce it
MIT License. See LICENSE for details.
Joao Felipe De Souza 2026