graceful-degradation-bench

Discrete-event simulation benchmark for graceful degradation strategies in LLM serving.

When a server is overloaded, rejection is the last resort. Before rejecting, the system can degrade service progressively:

Level 0: full service
Level 1: reduce output budget (512 -> 256 tokens)
Level 2: disable chain-of-thought
Level 3: downgrade to small model (7B -> 1.5B)
Level 4: selective drop of best-effort requests

Which degradation ladder policy maximizes served requests under quality and SLO constraints across different pressure regimes?

---

Why This Exists

Five prior benchmarks measured what happens under pressure but none measured the degradation strategy:

• multi-tenant-serving-bench: SLOs by tenant, but only admits or rejects
• admission-pressure-predictor: detects pressure before the cliff
• model-routing-complexity-bench: routes to smaller model in normal conditions
• slo-aware-autoscaling-sim: scales instances (slow, 5-60 seconds)
• kv-cache-aware-scheduler: rejects requests that do not fit

All of the above reject when capacity is exceeded. In production, rejection is the last resort. Before rejecting, the system can degrade progressively.

---

Key Results

Degrade early beats reject late

Aggressive early degradation outperforms rejection-only policies by 16-38% in composite serving score across all workloads.

aggressive_early is the best deployable policy in all workloads

Workload	Best deployable	Score	Second	Gap
premium_heavy	aggressive_early	76.9	fixed_threshold_ladder	7.6
ramp_to_overload	aggressive_early	88.2	fixed_threshold_ladder	3.3
spike_then_recover	aggressive_early	85.9	fixed_threshold_ladder	3.4
steady_moderate	aggressive_early	84.3	fixed_threshold_ladder	4.8
sustained_overload	aggressive_early	67.5	fixed_threshold_ladder	7.3

The gap is never a near-tie. Degrading early is robustly better.

Oracle wins under sustained and premium-heavy pressure

Among all policies including oracle:

• sustained_overload and premium_heavy: oracle_optimal wins
• ramp, spike, steady: aggressive_early wins

Oracle gains come from better quality preservation, not from serving more requests.

Graceful degradation creates SLO headroom

Premium tier SLO headroom (SLO = 1,200ms):

Policy	Premium headroom	Standard headroom
no_degradation	399 ms	1,706 ms
conservative_late	469 ms	1,754 ms
fixed_threshold_ladder	615 ms	1,922 ms
aggressive_early	660 ms	1,938 ms
oracle_optimal	673 ms	2,003 ms

Aggressive early degradation creates 65% more premium SLO margin than no_degradation, converting pressure into buffer rather than violations.

All degradation policies satisfy SLO even under sustained overload

The mechanism is explicit: degradation reduces per-request service time, keeping latency inside SLO boundaries even when the server is overloaded.

---

Degradation Policies

Policy	Kind	Strategy
no_degradation	baseline	reject when over capacity
conservative_late	ladder	degrade only at extreme pressure
fixed_threshold_ladder	ladder	fixed pressure thresholds per level
aggressive_early	ladder	lower thresholds, act before pressure cliff
tenant_aware_ladder	ladder	fixed thresholds, protects premium from levels 3-4
oracle_optimal	oracle	uses future pressure knowledge

---

Workloads

Workload	Description
steady_moderate	constant moderate pressure
ramp_to_overload	pressure ramps from 40% to 140%
spike_then_recover	pressure spike in the middle, then recovery
sustained_overload	constant high pressure
premium_heavy	50% premium tenants, ramp to overload

---

Quick Start

cd ~/dev/graceful-degradation-bench

python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

python -u run.py

Runtime: approximately 1-2 minutes. No GPU required.

---

Output Files

results/
  summary.csv
  decision_map.csv
  best_deployable.csv
  slo_headroom.csv

plots/
  01_pareto_goodput_slo.png
  02_goodput_score.png
  03_degradation_distribution.png
  04_tier_slo.png
  05_oracle_vs_aggressive.png
  06_best_deployable.png
  07_slo_headroom.png

---

Project Structure

graceful-degradation-bench/
+-- src/
|   +-- config.py
|   +-- workload.py
|   +-- server.py
|   +-- bench.py
|   +-- analysis.py
+-- results/
+-- plots/
+-- run.py
+-- README.md
+-- summary.txt
+-- design.md
+-- LICENSE
+-- requirements.txt

---

Methodology

This project is a calibrated discrete-event simulation, not an end-to-end serving benchmark.

Pressure is measured as:

pressure = max(kv_frac, batch_frac) + 0.15 * queue_frac

Each policy compares current pressure to its thresholds and selects a degradation level. Degradation reduces per-request cost and service time, freeing capacity for other requests.

The oracle policy uses lookahead pressure knowledge:

effective_pressure = max(current_pressure, mean(future_pressure[next_N]))

For full design details, see design.md.

---

Requirements

• Python 3.10+
• NumPy >= 1.26.0
• Pandas >= 2.0.0
• Matplotlib >= 3.8.0

No GPU required.

---

Practical Recommendation

If you need a simple default degradation policy:

• use aggressive_early as the primary strategy
• use fixed_threshold_ladder as a more conservative fallback
• avoid waiting for extreme pressure before engaging degradation
• monitor quality_adjusted_goodput, not raw goodput
• monitor premium SLO headroom as the primary safety signal

---

Documentation

• design.md -- detailed design rationale and modeling assumptions
• summary.txt -- concise high-level summary of findings
• LICENSE -- MIT License

---

License

MIT License -- Copyright (c) 2026 Joao Felipe De Souza

---

Author

Joao Felipe De Souza 2026