Skip to content

JohnScheuer/llm-serving-sim

Repository files navigation

llm-serving-sim

C++20 CMake Python License: MIT Baseline Sweep Pressure Sweep

End-to-end LLM serving simulator integrating scheduling, prefix caching, tensor allocation, and KV-cache management.

For detailed architecture, algorithm descriptions, and design rationale see DESIGN.md.


Portfolio Context

Final project in a five-project series modeling the full memory stack of an LLM server:

Project Focus Key Finding
kv-cache-compaction-lab KV-cache page compaction ThresholdCompaction dominates; 11 free-compaction points
prefix-cache-sim Prefix sharing with RadixTree LFU dominates small cache; multi-turn hit rate 60%+
llm-inference-scheduler Continuous batching ChunkedPrefill eliminates starvation; FCFS collapses
tensor-memory-allocator GPU tensor allocation Free-list beats buddy/slab for continuous size distributions
llm-serving-sim End-to-end integration ChunkedPrefill + LFU cache: 41% lower TTFT, 94% hit rate

"I implemented the memory layer of an LLM server from scratch -- from tensor allocator to inference scheduler -- then integrated all four components into a single end-to-end simulator."


Problem

Each component of an LLM serving system creates tradeoffs that only appear when the components interact:

  • A scheduler that minimizes TTFT admits requests aggressively -- and hits OOM first
  • A prefix cache with high hit rate reduces KV pressure -- but requires memory of its own
  • A tensor allocator with low fragmentation may have higher lookup cost -- affecting batch latency
  • KV page allocation grows per decode step -- competing with tensor scratch space

This project measures those interactions directly.


Architecture

Client requests (Poisson arrival, Zipf prefix distribution)
          |
          v
+-------------------+        +-------------------+
| IScheduler         |        | IPrefixCache       |
| FCFS / SJF /       | -----> | LFU / LRU / None   |
| ChunkedPrefill     |        | exact-match        |
+--------+----------+        +--------+----------+
         |                            |
         v                            v
+-------------------------------------------+
| MemoryManager                              |
|  IAllocator (tensor): first_fit /          |
|                       lifetime_aware       |
|  IKVCacheManager:     page-based           |
+--------+----------------------------------+
         |
         v
+-------------------+        +-------------------+
| ExecutionEngine    |        | MetricsCollector   |
| prefill + decode   | -----> | TTFT / TBT /       |
| cost model         |        | throughput / OOM   |
+-------------------+        +-------------------+

Key Findings

1. ChunkedPrefill reduces TTFT p95 by 41%

By interleaving chunked prefill work with decode, ChunkedPrefill prevents long prompts from blocking decode progress:

TTFT p95 by scheduler (mean across all configs):
  chunked_prefill:  19.7 ms
  fcfs:             33.8 ms
  sjf:              33.8 ms

2. LFU prefix cache reduces TTFT p95 by 57%

With Zipf-distributed prompts (alpha=1.2, 64 groups), LFU achieves 94.2% hit rate. Cached prefill tokens skip computation entirely:

TTFT p95 by prefix cache policy:
  lfu/lru: 16.2 ms   (94.2% hit rate)
  none:    43.3 ms   ( 0.0% hit rate)

3. ChunkedPrefill is the first to OOM

The scheduler that minimizes latency is also the most aggressive in admitting requests. At 4GB GPU memory, ChunkedPrefill produced OOM events while FCFS did not -- across all arrival rates tested.

OOM events at 4GB GPU memory:
  chunked_prefill: 2 events (arrival rate 40 req/s)
  fcfs:            0 events

This is the latency-capacity tradeoff: the scheduler that wins on
TTFT loses first on memory pressure.

4. Allocators equivalent under available memory

first_fit and lifetime_aware produce identical TTFT and fragmentation (~819 KB) because tensor allocation is a small fraction of total GPU memory in these configs. Differentiation requires heavier tensor workloads where the region-separation benefit of lifetime_aware becomes visible.

5. Combined optimum

Best configuration: chunked_prefill + lfu + first_fit
  TTFT p95:       11.5 ms
  Throughput TPS: 862 tokens/s
  Prefix hit rate: 91.6%
  OOM events:      0 (at 8GB+)

Quick Start

# Build
cmake -S . -B build -G Ninja
cmake --build build -j

# Generate workload
python3 experiments/workload_gen.py \
    --arrival-rate 10 \
    --duration-sec 60 \
    --n-prompt-groups 64 \
    --zipf-alpha 1.2 \
    --output results/workload.csv

# Run single configuration
./build/llm_serving_sim \
    --workload results/workload.csv \
    --scheduler chunked_prefill \
    --alloc first_fit \
    --prefix-cache lfu \
    --arrival-rate 10 \
    --ts-out results/ts.csv \
    --summary-out results/summary.csv

# Baseline sweep (72 runs)
python3 experiments/sweep_server.py

# Pressure sweep (96 runs, tight memory configs)
python3 experiments/sweep_pressure.py

# Plots
python3 plots/plot_server.py

# Analysis
python3 scripts/analyze_server.py
python3 scripts/analyze_pressure.py

CLI Reference

./build/llm_serving_sim [options]

  --workload FILE           CSV workload file (required)
  --scheduler STR           fcfs | sjf | chunked_prefill
  --alloc STR               first_fit | lifetime_aware
  --prefix-cache STR        none | lfu | lru
  --arrival-rate F          logical arrival rate label (written to summary CSV)
  --gpu-memory-mb N         total GPU memory (default: 40960)
  --kv-cache-mb N           KV cache pool (default: 20480)
  --tensor-pool-mb N        tensor allocator pool (default: 16384)
  --prefix-cache-mb N       prefix cache capacity (default: 4096)
  --max-batch-size N        max requests per batch (default: 64)
  --chunk-size N            prefill chunk tokens for ChunkedPrefill (default: 512)
  --short-frac F            SHORT region fraction for lifetime_aware (default: 0.5)
  --medium-frac F           MEDIUM region fraction for lifetime_aware (default: 0.3)
  --sim-duration-sec N      simulation duration fallback (default: 60)
  --ts-out FILE             time-series CSV output
  --summary-out FILE        summary CSV output (append mode)

Project Structure

llm-serving-sim/
|-- include/
|   |-- config.hpp              ServerConfig (all parameters)
|   |-- request.hpp             Request, RequestState, Priority
|   |-- scheduler.hpp           IScheduler + FCFS, SJF, ChunkedPrefill
|   |-- prefix_cache.hpp        IPrefixCache + LFU, LRU, None
|   |-- tensor_alloc.hpp        IAllocator + FreeList, LifetimeAware
|   |-- kv_cache.hpp            IKVCacheManager + page-based impl
|   |-- memory_manager.hpp      MemoryManager (coordinates tensor + KV)
|   |-- execution_engine.hpp    ExecutionEngine (prefill/decode cost)
|   |-- metrics.hpp             MetricsCollector, ServerMetrics
|   +-- simulator.hpp           LLMServerSimulator (main loop)
|-- src/
|   |-- main.cpp                CLI entry point
|   |-- scheduler.cpp           FCFS, SJF, ChunkedPrefill implementations
|   |-- prefix_cache.cpp        LFU/LRU/None prefix cache
|   |-- tensor_alloc.cpp        FreeList + LifetimeAware allocators
|   |-- kv_cache.cpp            Page-based KV cache manager
|   |-- memory_manager.cpp      admit_prefill, admit_decode, release
|   |-- execution_engine.cpp    prefill_seconds, decode_seconds
|   |-- metrics.cpp             TTFT, TBT, throughput, OOM collection
|   +-- simulator.cpp           Main event loop + CSV output
|-- experiments/
|   |-- workload_gen.py         Poisson + Zipf workload generator
|   |-- sweep_server.py         Baseline sweep (72 runs)
|   +-- sweep_pressure.py       Pressure sweep (96 runs)
|-- plots/
|   +-- plot_server.py          5 plots: TTFT, throughput, hit rate, Pareto, breakdown
|-- scripts/
|   |-- analyze_server.py       Baseline sweep analysis
|   +-- analyze_pressure.py     Pressure sweep: OOM, fragmentation
|-- results/
|   |-- sweep_summary.csv       72-row baseline results
|   |-- pressure_summary.csv    96-row pressure results
|   +-- plots/
|       |-- 01_ttft_p95_vs_arrival.png
|       |-- 02_throughput_tps_vs_arrival.png
|       |-- 03_prefix_hit_rate_vs_arrival.png
|       |-- 04_pareto_ttft_vs_throughput.png
|       +-- 05_time_breakdown_by_scheduler.png
|-- DESIGN.md                   Detailed architecture and rationale
|-- LICENSE                     MIT
+-- README.md

About

End-to-end LLM serving simulator integrating scheduling, prefix caching, tensor allocation, and KV-cache management. 168-run sweep (72 baseline + 96 pressure). Key finding: ChunkedPrefill + LFU cache achieves 41% lower TTFT p95 and 94% prefix hit rate, but hits OOM first under memory pressure.

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages