Analytical benchmark connecting pipeline parallelism and KV cache management in LLM serving.
With PP=4, each stage holds n_layers/4 layers and an independent KV cache. This changes disaggregation cost, prefix sharing efficiency, and tiering overhead in ways that are not obvious from first principles.
How does pipeline parallelism change the economics of KV cache management?
The portfolio has tensor-parallel-kv-cache-bench for TP. The natural gap is PP:
- comm-cost-modeling: PP has 46.7% bubble time at PP=8
- tensor-parallel-kv-cache-bench: TP shards KV by heads
- pipeline-parallel-kv-bench: PP shards KV by layers
In PP, each stage's KV is independent and complete — different from TP.
On NVLink, KV disaggregation can be fully hidden in bubble time
| Model | NVLink min PP for zero net cost | InfiniBand min PP | PCIe |
|---|---|---|---|
| llama3_70b | PP=4 | PP=8 | never |
| llama3_8b | PP=4 | PP=8 | never |
| mixtral_8x7b | PP=2 | PP=8 | never |
On NVLink, the KV transfer fits within pipeline bubble time. On PCIe, only 10-26% of transfer is absorbed by bubble even at PP=8.
A stage hit is only useful if all earlier stages also hit. This degrades prefix sharing savings sharply with PP degree:
| PP degree | Savings at 50% per-stage hit rate |
|---|---|
| 1 | 50.0% |
| 2 | 37.5% |
| 4 | 23.4% |
| 8 | 12.5% |
| PP | Hit rate | Independent model | Contiguous model | Overestimate |
|---|---|---|---|---|
| 4 | 25% | 25.0% | 8.3% | 3.0x |
| 8 | 25% | 25.0% | 4.2% | 6.0x |
| 8 | 50% | 50.0% | 12.5% | 4.0x |
This is the prefix reuse tax of pipeline parallelism.
Coordination overhead dominates at high PP for smaller models. PP=8 can reverse the speedup for llama3_8b and mixtral_8x7b. Optimal tiering PP is 4 for llama3_70b.
| Setup | Recommendation |
|---|---|
| NVLink + large models (70b+) | PP=4 is the sweet spot for zero disagg cost |
| NVLink + smaller models | PP=2-4, verify bubble covers transfer |
| InfiniBand | PP=8 for zero disagg cost; PP=2-4 with caution |
| PCIe | avoid PP-based KV disaggregation |
| Prefix sharing critical | prefer PP=1-2; PP=4+ imposes large reuse tax |
| Tiering | PP=4 optimal; PP=8 may reverse gain for small models |
| Model | Layers | batch_step_ms |
|---|---|---|
| llama3_8b | 32 | 5.0 ms |
| llama3_70b | 80 | 22.0 ms |
| mixtral_8x7b | 32 | 12.0 ms |
| Interconnect | Bandwidth |
|---|---|
| NVLink | 600 GB/s |
| InfiniBand 400G | 50 GB/s |
| PCIe gen4 | 3.1 GB/s |
cd ~/dev/pipeline-parallel-kv-bench
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
python -u run.py
Runtime: approximately 10 seconds. No GPU required.
results/
kv_layout.csv
disagg_analysis.csv
prefix_sharing.csv
tiering_analysis.csv
bubble_hidden_thresholds.csv
prefix_breakdown.csv
plots/
01_kv_per_stage.png
02_disagg_cost.png
03_bubble_overlap.png
04_prefix_sharing.png
05_tiering_speedup.png
06_prefix_degradation.png
07_bubble_thresholds.png
08_prefix_overestimate.png
This project is an analytical model calibrated from prior benchmarks.
Key models:
Bubble overlap: net_cost = max(0, transfer_ms - bubble_available_ms)
Contiguous prefix: savings = h * (1 - h^PP) / (PP * (1 - h))
Tiering parallel: cost = per_stage_ms + coordination_ms
For full design details, see design.md.
- Python 3.10+
- NumPy >= 1.26.0
- Pandas >= 2.0.0
- Matplotlib >= 3.8.0
No GPU required.
- design.md -- detailed design rationale and model equations
- summary.txt -- concise high-level summary of findings
- LICENSE -- MIT License
MIT License -- Copyright (c) 2026 Joao Felipe De Souza
Joao Felipe De Souza 2026