[Auto-FL] Harden cleanup and comparison budgets

[holgerroth]

Summary

This draft proposes two general Auto-FL runner improvements based on the recent long-running H100 campaigns:

• terminate every Linux process that belongs to one trial, including simulator workers that detach into a new process group or session;
• add an optional, task-defined observed comparison-budget contract for strict campaigns.

The default Auto-FL workflow is unchanged when a task does not configure observed budget evidence.

Trial process ownership

Each runner execution now injects a random NVFLARE_AUTOFL_TRIAL_TOKEN into the child environment. Existing process-group cleanup remains the portable first line of defense. On Linux, timeout, interruption, and final cleanup also scan /proc/*/environ for the exact token and terminate remaining trial-owned processes.

This avoids command-name, candidate-name, or shared-directory matching and directly covers simulator workers that create their own process groups.

Observed comparison budgets

Static import and fixed arguments can constrain declared budgets, but they cannot prove the behavior of arbitrary Python training loops. A task that needs strict semantic equality can now declare:

comparison_budget_evidence:
  artifact: server/autofl_budget_summary.json
  fields:
    - optimizer_steps
    - examples_seen
    - partition_sha256

The job emits:

{
  "values": {
    "optimizer_steps": 1280,
    "examples_seen": 81920,
    "partition_sha256": "..."
  }
}

The runner copies the contract into autofl.yaml, stores the scored baseline values, and requires exact evidence equality for simulation and externally recorded POC/production candidates. Missing, malformed, or different evidence makes the candidate a failed attempt and prevents score retention. Evidence and provenance are recorded in the candidate manifest, campaign metadata, results.tsv, and the running report.

The fields remain task-defined so this mechanism does not encode CIFAR, PyTorch, DataLoader, or optimizer-specific assumptions.

Review focus

• Is inherited per-trial process ownership the right minimal fix for detached simulator workers?
• Is opt-in job-emitted evidence the right boundary for strict compute comparability?
• Should the companion report-skill PR consume the new ledger evidence before this leaves draft?

Out of scope

• inferring optimizer-step equality from source inspection;
• changing seed/search policy;
• restructuring literature-review provenance;
• adding H100-specific code, fixtures, or instructions.

Validation

• 333 passed, 1 skipped across focused Auto-FL runner, guard, importer, plotter, report, and trigger-eval suites;
• Auto-FL runner suite: 157 passed, 1 skipped;
• skill admission: 0 errors, 0 warnings;
• Black, isort, flake8, and git diff --check: pass;
• docs build: pass (repository's existing 123 warnings);
• SKILL.md: 200/200 lines.

The skipped regression is Linux-specific and exercises a detached child that starts a new session; it will run in Linux CI. A broader local agent-skill sweep also reached 150 passing tests, with one unrelated macOS XGBoost import failure because libomp.dylib is unavailable.

[holgerroth]

/build

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 64.26%. Comparing base (8b9541c) to head (b3056ca).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #4984   +/-   ##
=======================================
  Coverage   64.26%   64.26%           
=======================================
  Files        1030     1030           
  Lines      103578   103578           
=======================================
+ Hits        66560    66562    +2     
+ Misses      37018    37016    -2     

Flag	Coverage Δ
unit-tests	64.26% <ø> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.