Author: Joao Felipe De Souza Year: 2026
This project simulates canary deployment strategies for LLM model updates.
The central question is:
Which rollout policy best balances safety and deployment progress when replacing one LLM version with another in production?
This is a deployment-safety benchmark, not a model accuracy benchmark.
The portfolio covers the full serving lifecycle except model replacement:
- model-routing-complexity-bench: routes by difficulty, same model version
- serving-cost-model-v2: cost differences between model sizes
- slo-aware-autoscaling-sim: capacity during transitions
- graceful-degradation-bench: fallback if quality degrades under pressure
- model-ab-testing-sim: safe rollout of new model versions
In production, replacing a model is risky. A new checkpoint may degrade quality, increase latency, or raise error rates. Canary deployment mitigates this by routing a small fraction of traffic to the candidate model before full rollout.
Replace baseline immediately. No guardrails. Baseline anti-pattern.
Never roll out the candidate. Zero deployment value. Safety ceiling only.
Route 1 percent of traffic to candidate indefinitely. Check guardrails every MIN_SAMPLE_SIZE candidate requests. Roll back and hold at 0 percent if guardrails fail. Cannot complete rollout.
Same as fixed_canary_1pct but at 5 percent. Slightly faster signal, slightly more exposure.
Route through stages: 1, 5, 10, 25, 50, 100 percent. Check guardrails at each stage boundary. On failure: roll back one stage and retry. On success: advance to next stage. Can complete full rollout.
Same stages as progressive_canary. On failure: hard stop, permanently hold at 0 percent, no retry. Safer under bad candidates.
Candidate is strictly better on all dimensions:
- faster decode
- higher quality
- lower error rate
- lower cost
Expected outcome: rollout should complete.
Candidate has quality_base = 0.860 vs baseline 0.920. Error rate increased from 0.005 to 0.012.
Expected outcome: rollback should trigger.
Candidate decode_ms_per_token = 3.40 vs baseline 2.20 (55% slower).
Expected outcome: rollback should trigger on latency guardrail.
Candidate is slightly worse on all dimensions but below all guardrail thresholds.
Expected outcome: policy decision depends on guardrail calibration.
quality_drop_max: 0.03 latency_increase_frac_max: 0.15 error_rate_increase_max: 0.005 cost_increase_frac_max: 0.10
These are fixed and not adapted during rollout. Guardrail calibration is the most important deployment configuration decision.
- bad_exposure_frac: fraction of requests served by bad candidate
- rolled_back: whether rollback was triggered
- rollback_count: how many times rollback occurred
- hard_stopped: whether strict canary permanently halted
- fully_rolled_out: whether candidate reached 100 percent traffic
- full_rollout_at_request: how many requests until full rollout
- exposure_to_candidate: total fraction routed to candidate
- correct_safety_decision: did the policy make the right decision?
- rollout_value_score: composite score for rollout-capable policies
- quality_delta: candidate quality minus baseline quality
- latency_delta_frac: relative latency change
- error_rate_delta: absolute error rate change
This benchmark separates two distinct objectives:
Safety objective: Which policy minimizes bad user exposure? Answer: never roll out anything. Practical answer: fixed canary at 1-5 percent.
Deployment objective: Which rollout-capable policy best balances safety and progress? Answer: fixed canary for bad candidates, progressive canary for good candidates.
This dual framing is the core contribution of the benchmark.
Fixed canary: monitoring strategy, containment, never promotes Progressive canary: deployment strategy, promotes good candidates Strict progressive canary: safest deployment strategy, no retry under regression
- model-routing-complexity-bench: decision of which model per request
- serving-cost-model-v2: cost of the new model vs old
- graceful-degradation-bench: fallback if new model degrades
- model-ab-testing-sim: how to safely roll out the new version
This is a calibrated simulation, not a live serving system.
Limitations:
- synthetic traffic distributions
- proxy quality scores rather than measured task accuracy
- fixed guardrail thresholds
- no concept of user sessions or cohorts
- single request population, no differentiation by request type
- no feedback loop between candidate quality and traffic routing
These are acceptable because the benchmark targets relative comparison of rollout strategies, not absolute production safety guarantees.
This benchmark answers:
Not how to build a model, but how to safely replace one in production.
The key finding is that monitoring and deployment are different objectives that require different strategies. Fixed canary is best at containing regressions. Progressive canary is best at completing good rollouts. Strict progressive canary is the safest strategy that still enables deployment progress.
Author: Joao Felipe De Souza Year: 2026