Skip to content

Tron api returns 500 errors on jobruncollection mutations during reads - #1114

Open
CharlesPeacock wants to merge 5 commits into
masterfrom
TRON-2683_Tron_api_returns_500_errors_on_jobruncollection_mutations_during_reads
Open

Tron api returns 500 errors on jobruncollection mutations during reads#1114
CharlesPeacock wants to merge 5 commits into
masterfrom
TRON-2683_Tron_api_returns_500_errors_on_jobruncollection_mutations_during_reads

Conversation

@CharlesPeacock

@CharlesPeacock CharlesPeacock commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

This PR is a follow-up to PR-1113 and changes should be reviewed together (Or 1113 can be reviewed first, the branch for this PR is based off of 1113).
This PR adds fixes to a recently discovered race condition inside of JobRunCollection methods. When handling concurrent API requests, /api/jobs reads from a shared deque inside JobRunCollection, this deque can be mutated during reads by the reactor, resulting in replicable 500 errors with RuntimeError: deque mutated during iteration.

I fixed the issue by adding a snapshot of the deque to every method inside of JobRunCollection that reads from it. Since this operation is atomic, it ensures reads stay consistent and we don't read from a mutating deque.

Aside from passing tests, I made a small script that makes concurrent API calls to the Tron api, which I initially used to discover the issue.
After performing the change, there were no errors during API testing (for this issue specifically, errors would come out in phase 3 of this script).
Keep in mind that this script was made to test the previous changes, related to api GET endpoints moving from @exclusive to @bounded.
Sample output of script:

=== 0. Connectivity check ===

  Connected! Got 200 from http://localhost:8089/api/status

=== 1. Endpoints return valid responses ===
GET /api/metrics ... OK (200)
  Response body valid ... OK
GET /api/events ... OK (200)
  Response body valid ... OK
GET /api/status (baseline) ... OK (200)

=== 2. Concurrency test: metrics/events no longer block other reads ===
Firing GET /api/metrics and GET /api/status simultaneously...
If metrics were still exclusive, status would have to wait.

  [started: 2026-08-18 09:01:07,335]
  /api/status  returned in 15ms
  /api/metrics returned in 17ms
  [ended:   2026-08-18 09:01:07,355]
  OK — status returned fast (15ms)

=== 3. Parallel read stress test: 40 concurrent GETs ===
Testing for RuntimeError / crashes under concurrent dict iteration...

  [started: 2026-08-18 09:01:07,356]
  [ended:   2026-08-18 09:01:07,422]
  OK — all 40 requests succeeded with no errors

=== 3b. Lock contention: metrics reads while generating new metric keys ===

  [started: 2026-08-18 09:01:07,423]
  [ended:   2026-08-18 09:01:50,105]
  OK — all 40 requests succeeded under lock contention

=== 3c. Events mutation race: GET events while discarding events ===
Tests dict(event_log) copy safety against direct concurrent deletion...

  [started: 2026-08-18 09:01:50,107]
  Seeding 20 unique events... enqueued
  [ended:   2026-08-18 09:02:01,892]
  OK — concurrent event read/write safe

=== 4. Sustained concurrent load: readers + writers overlapping ===
Firing reads while POSTing to trigger metric writes...

  [started: 2026-08-18 09:02:01,893]
  [ended:   2026-08-18 09:02:21,563]
  OK — no failures during concurrent read/write

=== 5. Check trond logs for relevant exceptions ===
(Ignoring kubernetes/task_processing noise from local dev)
  OK — no relevant exceptions in trond log

@CharlesPeacock
CharlesPeacock requested a review from a team as a code owner August 18, 2026 16:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant