An Efficient and Versatile Inference Engine for Distributed LLM Serving
A lightweight, educational LLM inference engine for studying continuous batching, paged KV cache, chunked prefill, and online serving.
Real PyTorch inference server comparing FCFS, EagerContBatch, and ChunkedPrefill on RTX 2070. Key finding: ChunkedPrefill is not a throughput optimizer but a fairness mechanism — it reduces worst-case decode stalls from 58.6ms to 34.1ms while EagerContBatch maximizes mean TTFT and throughput.
Reproducible vLLM serving scheduler benchmark with 144-run cross-layer performance analysis.
Long-context benchmark pushing Qwen2-0.5B from 4K to 32K tokens on RTX 2070 using SDPA + chunked prefill. Shows 40x speedup at 8K, FP16 beating INT4 at long context, and that quantization is NOT a long-context solution — KV-cache is the real bottleneck.
Step-level simulation of chunked prefill under KV cache pressure. Block policy wastes 89.6% of prefill chunks while achieving the same throughput as upfront rejection. Chunk waste rate is the metric standard throughput metrics miss.
A minimal, native Metal inference engine for Qwen3-30B-A3B on Apple Silicon Macs.
Add a description, image, and links to the chunked-prefill topic page so that developers can more easily learn about it.
To associate your repository with the chunked-prefill topic, visit your repo's landing page and select "manage topics."