End-to-end LLM serving simulator integrating scheduling, prefix caching, tensor allocation, and KV-cache management.
For detailed architecture, algorithm descriptions, and design rationale see DESIGN.md.
Final project in a five-project series modeling the full memory stack of an LLM server:
| Project | Focus | Key Finding |
|---|---|---|
| kv-cache-compaction-lab | KV-cache page compaction | ThresholdCompaction dominates; 11 free-compaction points |
| prefix-cache-sim | Prefix sharing with RadixTree | LFU dominates small cache; multi-turn hit rate 60%+ |
| llm-inference-scheduler | Continuous batching | ChunkedPrefill eliminates starvation; FCFS collapses |
| tensor-memory-allocator | GPU tensor allocation | Free-list beats buddy/slab for continuous size distributions |
| llm-serving-sim | End-to-end integration | ChunkedPrefill + LFU cache: 41% lower TTFT, 94% hit rate |
"I implemented the memory layer of an LLM server from scratch -- from tensor allocator to inference scheduler -- then integrated all four components into a single end-to-end simulator."
Each component of an LLM serving system creates tradeoffs that only appear when the components interact:
- A scheduler that minimizes TTFT admits requests aggressively -- and hits OOM first
- A prefix cache with high hit rate reduces KV pressure -- but requires memory of its own
- A tensor allocator with low fragmentation may have higher lookup cost -- affecting batch latency
- KV page allocation grows per decode step -- competing with tensor scratch space
This project measures those interactions directly.
Client requests (Poisson arrival, Zipf prefix distribution)
|
v
+-------------------+ +-------------------+
| IScheduler | | IPrefixCache |
| FCFS / SJF / | -----> | LFU / LRU / None |
| ChunkedPrefill | | exact-match |
+--------+----------+ +--------+----------+
| |
v v
+-------------------------------------------+
| MemoryManager |
| IAllocator (tensor): first_fit / |
| lifetime_aware |
| IKVCacheManager: page-based |
+--------+----------------------------------+
|
v
+-------------------+ +-------------------+
| ExecutionEngine | | MetricsCollector |
| prefill + decode | -----> | TTFT / TBT / |
| cost model | | throughput / OOM |
+-------------------+ +-------------------+
By interleaving chunked prefill work with decode, ChunkedPrefill prevents long prompts from blocking decode progress:
TTFT p95 by scheduler (mean across all configs):
chunked_prefill: 19.7 ms
fcfs: 33.8 ms
sjf: 33.8 ms
With Zipf-distributed prompts (alpha=1.2, 64 groups), LFU achieves 94.2% hit rate. Cached prefill tokens skip computation entirely:
TTFT p95 by prefix cache policy:
lfu/lru: 16.2 ms (94.2% hit rate)
none: 43.3 ms ( 0.0% hit rate)
The scheduler that minimizes latency is also the most aggressive in admitting requests. At 4GB GPU memory, ChunkedPrefill produced OOM events while FCFS did not -- across all arrival rates tested.
OOM events at 4GB GPU memory:
chunked_prefill: 2 events (arrival rate 40 req/s)
fcfs: 0 events
This is the latency-capacity tradeoff: the scheduler that wins on
TTFT loses first on memory pressure.
first_fit and lifetime_aware produce identical TTFT and fragmentation (~819 KB) because tensor allocation is a small fraction of total GPU memory in these configs. Differentiation requires heavier tensor workloads where the region-separation benefit of lifetime_aware becomes visible.
Best configuration: chunked_prefill + lfu + first_fit
TTFT p95: 11.5 ms
Throughput TPS: 862 tokens/s
Prefix hit rate: 91.6%
OOM events: 0 (at 8GB+)
# Build
cmake -S . -B build -G Ninja
cmake --build build -j
# Generate workload
python3 experiments/workload_gen.py \
--arrival-rate 10 \
--duration-sec 60 \
--n-prompt-groups 64 \
--zipf-alpha 1.2 \
--output results/workload.csv
# Run single configuration
./build/llm_serving_sim \
--workload results/workload.csv \
--scheduler chunked_prefill \
--alloc first_fit \
--prefix-cache lfu \
--arrival-rate 10 \
--ts-out results/ts.csv \
--summary-out results/summary.csv
# Baseline sweep (72 runs)
python3 experiments/sweep_server.py
# Pressure sweep (96 runs, tight memory configs)
python3 experiments/sweep_pressure.py
# Plots
python3 plots/plot_server.py
# Analysis
python3 scripts/analyze_server.py
python3 scripts/analyze_pressure.py
./build/llm_serving_sim [options]
--workload FILE CSV workload file (required)
--scheduler STR fcfs | sjf | chunked_prefill
--alloc STR first_fit | lifetime_aware
--prefix-cache STR none | lfu | lru
--arrival-rate F logical arrival rate label (written to summary CSV)
--gpu-memory-mb N total GPU memory (default: 40960)
--kv-cache-mb N KV cache pool (default: 20480)
--tensor-pool-mb N tensor allocator pool (default: 16384)
--prefix-cache-mb N prefix cache capacity (default: 4096)
--max-batch-size N max requests per batch (default: 64)
--chunk-size N prefill chunk tokens for ChunkedPrefill (default: 512)
--short-frac F SHORT region fraction for lifetime_aware (default: 0.5)
--medium-frac F MEDIUM region fraction for lifetime_aware (default: 0.3)
--sim-duration-sec N simulation duration fallback (default: 60)
--ts-out FILE time-series CSV output
--summary-out FILE summary CSV output (append mode)
llm-serving-sim/
|-- include/
| |-- config.hpp ServerConfig (all parameters)
| |-- request.hpp Request, RequestState, Priority
| |-- scheduler.hpp IScheduler + FCFS, SJF, ChunkedPrefill
| |-- prefix_cache.hpp IPrefixCache + LFU, LRU, None
| |-- tensor_alloc.hpp IAllocator + FreeList, LifetimeAware
| |-- kv_cache.hpp IKVCacheManager + page-based impl
| |-- memory_manager.hpp MemoryManager (coordinates tensor + KV)
| |-- execution_engine.hpp ExecutionEngine (prefill/decode cost)
| |-- metrics.hpp MetricsCollector, ServerMetrics
| +-- simulator.hpp LLMServerSimulator (main loop)
|-- src/
| |-- main.cpp CLI entry point
| |-- scheduler.cpp FCFS, SJF, ChunkedPrefill implementations
| |-- prefix_cache.cpp LFU/LRU/None prefix cache
| |-- tensor_alloc.cpp FreeList + LifetimeAware allocators
| |-- kv_cache.cpp Page-based KV cache manager
| |-- memory_manager.cpp admit_prefill, admit_decode, release
| |-- execution_engine.cpp prefill_seconds, decode_seconds
| |-- metrics.cpp TTFT, TBT, throughput, OOM collection
| +-- simulator.cpp Main event loop + CSV output
|-- experiments/
| |-- workload_gen.py Poisson + Zipf workload generator
| |-- sweep_server.py Baseline sweep (72 runs)
| +-- sweep_pressure.py Pressure sweep (96 runs)
|-- plots/
| +-- plot_server.py 5 plots: TTFT, throughput, hit rate, Pareto, breakdown
|-- scripts/
| |-- analyze_server.py Baseline sweep analysis
| +-- analyze_pressure.py Pressure sweep: OOM, fragmentation
|-- results/
| |-- sweep_summary.csv 72-row baseline results
| |-- pressure_summary.csv 96-row pressure results
| +-- plots/
| |-- 01_ttft_p95_vs_arrival.png
| |-- 02_throughput_tps_vs_arrival.png
| |-- 03_prefix_hit_rate_vs_arrival.png
| |-- 04_pareto_ttft_vs_throughput.png
| +-- 05_time_breakdown_by_scheduler.png
|-- DESIGN.md Detailed architecture and rationale
|-- LICENSE MIT
+-- README.md