feat: add storage concurrency benchmark - #319
Open
kisernl wants to merge 7 commits into
Open
Conversation
Contributor License AgreementAll contributors are covered by a CLA. |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
kisernl
force-pushed
the
storage-concurrency-bench
branch
from
August 14, 2026 15:16
4d5f0df to
33f1142
Compare
kisernl
added a commit
that referenced
this pull request
Aug 14, 2026
Adopts three practices from the storage concurrency benchmark (#319), which faced the same problem of making levels comparable to each other. Report percentiles only when the samples back them. A p95 drawn from one observation is the median again, so printing both claimed knowledge the run did not have, and the score weighted that single number at 25% for the median plus 20% for the p95. Stats now carry their sample count, and consumers withhold a p95 under 20 samples rather than repeating the median. #319 does this by returning null for p99 below 1,000 samples. Measure latency per loop, not per action phase. A level's action phase covers as many loops as that level runs, so round wall clocks were never comparable between levels; per loop, one session's ten actions while the level's sessions run together, is. On replayed data the levels land within 2.6s to 3.2s of each other, where the round wall clocks spread from 4s to 63s. The leaderboard and the CI table now read Loop and Loop (p95), and no longer fall back to taskMs, since presenting a whole action phase under a per-loop heading would relabel the number rather than report it. Take samples from the workload instead of from repetition. Sessions repeat their loop per level, 20/4/2/1/1, which evens the budget out at 20 to 50 loop samples where c1 previously had one session, ten actions, and a success rate that could only be 0% or 100%. Repeating inside the session costs no extra browsers, which is the expensive part, and mirrors how #319 pushes 1,200 operations through a fixed pool. A level action budget bounds the cost for slow providers: notte averaged 3.6s per action, which would be 12 minutes for c1 alone. The budget is checked on loop boundaries so every session stops together and the level keeps its concurrency while it runs. Count concurrency instead of inferring it. sessionsAlive measures survival, so sessions taking turns reported the same number as sessions running together. A tracker now counts sessions in flight and records the peak per round, both for live sessions and for sessions running actions, and the round logs a line when a level never reached its own session count or exceeded it. On replayed kernel data the new Peak column reads 10/50, which is its account limit made visible rather than inferred from a regex over error strings. This is the metric that would have caught c1 running against 91 live sessions directly. A session now counts as successful when every action it attempted succeeded, rather than against a fixed total, so a session stopped by the action budget is not recorded as a provider failure. Covered by benchmarks/browser/concurrent-sampling.test.ts, 13 checks over the tracker, the budget boundary, loop chunking and the percentile gates, following the tsx test-script convention in #319. Validated end to end by replaying real rounds from run 31626532250 through the writer, merge and both SVGs: every level reports 18 to 48 loop samples, c10 withholds its p95 at 18 samples while c50 reports it at 48, and the peaks read N/N.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
This benchmark measures how storage providers behave as the number of simultaneous GET clients increases. It is intended to identify:
It is a closed-loop concurrency benchmark: each worker waits for its GET to complete before issuing the next GET. Throughput is therefore an observed output, not a fixed offered request rate.
Benchmark structure
SINGLE_PREFIXandSPREAD_64The initial operation budget is 1,200 per cell, producing enough measured samples for p99 latency. Lower operation counts are supported for smoke tests.
How concurrency is established and validated
The runner's
concurrency: 1is intentionally not the storage concurrency setting. It means the platform executes one cell task at a time so cells cannot overlap. Each cell creates its own internal pool with exactly the requested number of async workers:A shared operation counter assigns exactly the cell's operation budget across those workers. The benchmark increments an
activeRequestscounter immediately before eachstorage.download()call and decrements it infinally, somaxActiveRequestsmeasures the highest number of simultaneous storage requests observed. A c128 cell with at least 128 operations must reportmaxActiveRequests: 128; otherwise the cell does not exercise the requested concurrency.The benchmark reports
maxActiveRequestswith every cell and marks request validity separately from task completion. A task can complete successfully while its requests are all 404s; in that casesuccessRateis 0,notFoundRateis 1, andvalidis false. Smoke tests use at least 256 operations to ensure c128 can reach 128 active requests; production cells use 1,200 operations and retain 1,140 measured samples after warmup.This is target closed-loop concurrency, not fixed-RPS load. Workers maintain up to N in-flight requests and immediately replace completed requests, but a request completion and replacement can create a brief scheduling gap. The benchmark therefore validates the target with
maxActiveRequestsrather than claiming an exact constant offered rate.Corpus seeding
The benchmark does not seed during measurement. The new seed utility creates the deterministic corpus once per provider and is idempotent:
The workflow supports selecting providers, choosing the operation count, and optionally seeding before a run.
Measurements
Each cell reports:
Invalid cells, such as cells whose corpus objects are missing, are explicitly marked and penalized rather than presented as performance data.
Composite scoring
Each provider receives a 0-100 composite score averaged across its cells:
The score uses fixed absolute ceilings so adding or removing providers does not change existing scores. Success rate and invalid cells reduce the final score.
Results and GitHub Actions
The benchmark writes a combined result file to:
The workflow uploads that JSON, then a collect job generates:
storage-concurrency.md, including provider ranking and cell detailsstorage-concurrency.svg, showing composite scoresThe Markdown report is appended to the GitHub Actions job Summary and all report files are uploaded as artifacts.
Validation
--noEmitcheck passesCaveats