fix: eliminate rickshaw-settings.json.xz race condition when multiple engines start simultaneously

[k-rister]

Problem

When multiple engine containers start simultaneously and share a common directory, they all execute bootstrap scripts that SCP rickshaw-settings.json.xz from the controller to the shared directory at the same time. The concurrent SCP writes corrupt the file — one container's partial write is overwritten mid-stream by another container's SCP.

From issue #870 logs:

client-1: SCP rickshaw-settings.json.xz -> /shared-engines-dir/rickshaw-settings.json.xz at 16:14:39.001, completed 16:14:39.612
client-2: SCP rickshaw-settings.json.xz -> /shared-engines-dir/rickshaw-settings.json.xz at 16:14:39.153, completed 16:14:39.726

Both SCPs overlap. The "SCP succeeded" message reports the SCP's own exit code — it does not mean the file on disk is intact, only that the bytes were sent. The other container's concurrent write can corrupt the file between the SCP completing and the engine reading it.

Root causes:

• remotehosts endpoint: Multiple containers on the same host share /shared-engines-dir via bind mount
• kube endpoint: Multiple containers in the same pod share an emptyDir volume at /shared-engines-dir
• osp endpoint: Not affected (VMs don't share storage)

8-stream tests succeed because fewer concurrent SCPs make the race unlikely. 16-stream tests hit it consistently because 16 concurrent SCPs almost guarantee an overlap.

Solution

Move the rickshaw-settings.json.xz copy operation from bootstrap (runs in every container) to the endpoint scripts (run once before containers/pods start).

remotehosts endpoint fix

Copy the file once per remote host before any containers are launched using the existing SSH infrastructure.

From the issue recommendation:

The endpoint script (endpoints/remotehosts/remotehosts.py) already copies files to each remote host via SSH before containers start — it copies per-engine _env.txt files to each remote's cfg/ directory during the create_podman phase. The SSH connection and file copy infrastructure is proven and already in use.

kube endpoint fix

Use a Kubernetes initContainer that downloads the file once before the main containers in each pod start. The initContainer:

• Runs before any main containers
• Mounts the shared emptyDir volume
• Downloads rickshaw-settings.json.xz from the controller via SCP
• Uses printf '%b' to properly handle SSH key newlines
• Completes before main containers start

This is the Kubernetes-native way to prepare shared data for pods.

bootstrap.py fallback

Bootstrap now checks if the file already exists:

• If present (endpoint pre-copied it): Use it and log success
• If missing: Fall back to SCP with a warning (backwards compatibility)

This maintains compatibility with any endpoints that haven't been updated yet.

Changes

• endpoints/remotehosts/remotehosts.py:

  • Added copy_rickshaw_settings_worker_thread() - worker thread to copy settings file to a remote
  • Added copy_rickshaw_settings_to_remotes() - orchestrates copying to all remotes in parallel
  • Called the new function in main() after create_remote_dirs() and before remotes_pull_images()

• endpoints/kube/kube.py:

  • Added initContainer to each pod that downloads rickshaw-settings.json.xz before main containers start
  • InitContainer uses bash/scp to fetch the file from the controller
  • Uses printf '%b' to properly convert \n sequences to real newlines for SSH key
  • Writes to the shared emptyDir volume at /shared-engines-dir/

• engine/bootstrap.py:

  • Check if rickshaw-settings.json.xz already exists
  • If present, use it (endpoint pre-copied it)
  • If missing, fall back to SCP with warning message

Testing

remotehosts endpoint ✅

Ran a 4-engine fio test across 4 remote hosts:

• ✅ File successfully copied to all 4 remotes before container launch
• ✅ All engines found the file present (no SCP attempts from bootstrap)
• ✅ Test completed successfully (exit code 0)
• ✅ No file corruption observed

Endpoint log excerpt:

[LOG 2026-08-20 16:11:19,131 INFO remotehosts copy_rickshaw_settings_to_remotes:881][Thread Main] Copying rickshaw-settings.json.xz to remotes before container launch
[LOG 2026-08-20 16:11:19,296 INFO remotehosts copy_rickshaw_settings_worker_thread:855][Thread CRSWT-0] [Remote 192.168.12.212] Copied /var/lib/crucible/run/.../rickshaw-settings.json.xz to 192.168.12.212:/var/lib/crucible/remotehosts-1_.../data/rickshaw-settings.json.xz

Engine log excerpt:

2026-08-20 16:12:35,448 bootstrap INFO Found rickshaw-settings.json.xz at /shared-engines-dir/rickshaw-settings.json.xz (endpoint pre-copied it)

kube endpoint ✅

Ran a fio test on Kubernetes cluster:

• ✅ InitContainer completed successfully (exitCode: 0, reason: "Completed")
• ✅ Bootstrap found the pre-copied file (no SCP fallback)
• ✅ Test completed successfully (exit code 0)

Endpoint log excerpt:

{
  "exitCode": 0,
  "finishedAt": "2026-08-20T23:14:52Z",
  "reason": "Completed",
  "startedAt": "2026-08-20T23:14:52Z"
}

Engine log excerpt:

2026-08-20 23:14:55,043 bootstrap INFO Found rickshaw-settings.json.xz at /shared-engines-dir/rickshaw-settings.json.xz (endpoint pre-copied it)

Why This Fix is Correct

• No race condition possible: File is written before containers start (remotehosts) or by a single initContainer before main containers (kube)
• No wait loops, designated writers, or atomic rename tricks needed
• No bootstrap coordination required
• Bootstrap becomes simpler: Check first, SCP only as fallback
• Scales to any number of engines per host/pod
• Backwards compatible: Endpoints not yet updated still work via SCP fallback

Closes #870

🤖 Generated with Claude Code

[k-rister]

InitContainer Lifecycle Explanation

For reviewers unfamiliar with Kubernetes initContainers, here's how they solve the race condition:

Pod Startup Sequence

Pod Scheduled → Pull Images → Run initContainers → Run main containers

InitContainer Execution (Sequential, Blocking)

For each initContainer (in order):
  ├─ Pull container image
  ├─ Start container
  ├─ Wait for container to complete
  ├─ Check exit code
  │  ├─ exitCode 0 → Continue to next initContainer (or main containers)
  │  └─ exitCode != 0 → STOP! Pod enters "Init:Error" state
  └─ Container terminates (initContainers never stay running)

Our Specific Flow

What happens in our pod:

┌─────────────────────────────────────────────────────────┐
│ Pod: rickshaw-client-1                                  │
├─────────────────────────────────────────────────────────┤
│                                                         │
│ 1. initContainer: "copy-rickshaw-settings"             │
│    ├─ Image: quay.io/crucible/engines:...              │
│    ├─ Mounts: /shared-engines-dir (emptyDir)           │
│    ├─ Command: /bin/bash -c                            │
│    │   └─ Write SSH key to /tmp/ssh_id                 │
│    │   └─ SCP rickshaw-settings.json.xz from controller│
│    │   └─ Write to /shared-engines-dir/                │
│    ├─ Exit code: 0 ✅                                   │
│    └─ Container TERMINATES                             │
│                                                         │
│ ──────── initContainer done, main containers start ────│
│                                                         │
│ 2. Container: "client-1" (benchmark engine)            │
│    ├─ Image: quay.io/crucible/engines:...              │
│    ├─ Mounts: /shared-engines-dir (SAME emptyDir)      │
│    ├─ Starts bootstrap.py                              │
│    │   └─ Checks /shared-engines-dir/rickshaw-settings │
│    │   └─ File EXISTS ✅ (initContainer wrote it)      │
│    │   └─ Uses file, NO SCP needed                     │
│    └─ Runs benchmark                                   │
│                                                         │
│ 3. Container: "profiler-sysstat-1" (tool)              │
│    ├─ Mounts: /shared-engines-dir (SAME emptyDir)      │
│    ├─ File ALREADY exists from initContainer           │
│    └─ Uses file, NO SCP needed                         │
│                                                         │
│ (All containers share the SAME emptyDir volume)        │
└─────────────────────────────────────────────────────────┘

Key Properties of InitContainers

Sequential Execution:

• If you have multiple initContainers, they run one at a time in order
• Each must complete successfully before the next starts

Blocking:

• Main containers CANNOT start until ALL initContainers succeed
• This guarantees the file is present before any main container runs

Failure Handling:

If initContainer fails (exitCode != 0):
  ├─ Main containers NEVER start
  ├─ Pod status becomes "Init:Error"
  ├─ Kubernetes may retry based on restartPolicy
  └─ You see this in: oc get pods → STATUS: "Init:Error"

Shared Storage:

• InitContainers can mount the same volumes as main containers
• In our case: the emptyDir at /shared-engines-dir
• Files written by initContainer are visible to main containers

Why This Eliminates the Race

BEFORE (broken):

Main containers all start at ~same time:
  Container 1: SCP rickshaw-settings.json.xz ─┐
  Container 2: SCP rickshaw-settings.json.xz ─┼─→ RACE! Corruption!
  Container 3: SCP rickshaw-settings.json.xz ─┘
     (All writing to same file simultaneously)

AFTER (fixed):

Time 0: initContainer: SCP rickshaw-settings.json.xz
  └─→ Completes successfully, file written ✅
  
Time 1: initContainer exits (exitCode 0)
  
Time 2: Kubernetes starts ALL main containers
  ├─ Container 1: File already exists, use it ✅
  ├─ Container 2: File already exists, use it ✅  
  └─ Container 3: File already exists, use it ✅
     (NO concurrent writes - file was written BEFORE they started)

Evidence from Testing

InitContainer status (from endpoint logs):

{
  "containerID": "cri-o://8b9836c10021...",
  "exitCode": 0,
  "reason": "Completed",
  "startedAt": "2026-08-20T23:14:52Z",
  "finishedAt": "2026-08-20T23:14:52Z"
}

Main container bootstrap log:

2026-08-20 23:14:55,043 bootstrap INFO Found rickshaw-settings.json.xz at /shared-engines-dir/rickshaw-settings.json.xz (endpoint pre-copied it)

Notice the timestamps:

• 23:14:52 - initContainer finished
• 23:14:55 - Main container started (3 seconds later)

The main container literally couldn't start until the initContainer succeeded!

Summary

InitContainers are Kubernetes' way of saying:

"Do this setup work FIRST, and don't let anything else run until it's done successfully."

This guarantees our settings file is present before any engine containers try to use it, completely eliminating the race condition.

[k-rister]

PR Review: rickshaw#871 — fix: eliminate rickshaw-settings.json.xz race condition when multiple engines start simultaneously

Summary: Eliminates a race condition that corrupts rickshaw-settings.json.xz under high concurrency by shifting the file copy from the container's boot script to the endpoint scripts (running once before startup) and implementing an existence-check fallback in bootstrap.py.
Changed files: 3
Review dimensions: Correctness, API & Contracts, Build & Deploy, Documentation, Style, Completeness

Issues

• [endpoints/remotehosts/remotehosts.py:2658] Ignored return value — The return code rc from copy_rickshaw_settings_to_remotes() is ignored in main(). If copying the settings file to a remote fails during setup, the execution will proceed silently at the endpoint layer. (Though the container's bootstrap will still fall back to SCP with a warning at boot time).
• [endpoints/kube/kube.py] [endpoints/remotehosts/remotehosts.py] Untested code paths — The newly added functions and logic (e.g. copy_rickshaw_settings_to_remotes and initContainers generator block) do not have corresponding unit test coverage in tests/. (Note: This is consistent with other pre-existing endpoint functions which also lack unit tests).

File Coverage

• endpoints/kube/kube.py — No issues
• endpoints/remotehosts/remotehosts.py — 1 issue (ignored return value)
• engine/bootstrap.py — No issues

Limitations

• Runtime validation: Unable to execute live runtime testing on an active Kubernetes cluster or live remote hosts.
• Performance: Cannot verify network or file-system performance under massive multi-concurrency (e.g. 16+ streams) in a real physical environment.

Verdict

Approve with comments — The PR is incredibly well-designed, extremely clean, backwards-compatible, and resolves a highly critical race condition without needing complex coordination or locking. The ignored return code in remotehosts.py is minor and can be addressed at the author's discretion.

[k-rister]

Addressed the ignored-return-value finding: main() now captures copy_settings_rc from copy_rickshaw_settings_to_remotes() and logs an error if any remote copy fails. Left it non-fatal rather than aborting the run, since bootstrap.py's SCP fallback already covers this case for the affected engine(s).

The missing unit test coverage note was left as-is per the review's own observation that it's consistent with other pre-existing endpoint functions.

Commit: 6af45cb

[atheurer]

LGTM! The pre-copy strategy via Kubernetes initContainers and remotehosts thread pool cleanly eliminates the race condition while preserving the SCP fallback in bootstrap.py.