Integration harness exits 0 when workflow tests are silently skipped

[ericfitz]

The integration runner can skip its main test body and still report success.

What happened

A make test-integration run reported:

=== SUMMARY ===
  Integration tests: 22 passed, 0 failed, 1 skipped
  Packages: 4 passed, 0 failed

exit code 0. But test/integration/workflows — the package holding the actual end-to-end API tests — never ran. Buried above the summary:

[WARNING] Test server did not become ready (see logs/tmi-test-server.log) — skipping workflow tests

A healthy run of the same command executes 7 packages and 82 tests, including the workflows module (~537s).

Why exit 0

In scripts/run-integration-tests.py, workflow_exit is initialized to 0 and only assigned inside the branch that runs when the test server came up. When the server fails to start, the variable keeps its initial 0 and the final return api_exit if api_exit != 0 else workflow_exit yields success.

Why it matters

This is a gate. A silent skip that still exits 0 means CI (or a person) can believe the API contract was verified when it was not — the 22 tests that did run are the in-process api package, not the end-to-end workflows.

Suggested fix

Treat "server did not become ready" as a failure, or track a distinct workflows_skipped flag and make the summary and exit code reflect it. At minimum the summary line should not read like a clean pass when a whole package was skipped.

Root cause of the skip in this instance (separate, already addressed)

mkdir /tmp/tmi-test-logs: no space left on device inside the container — Docker's VM disk was full because test/results/ (one ~1.4 GB SQLite DB per CATS run, 14 GB total) was not in .dockerignore and was being transferred and COPYed into every build. Fixed separately; the harness behavior above is the issue here.

[ericfitz]

Fixed on dev/1.6.0/cats-fixes (unpushed).

workflow_exit could not express this: it was 0 before the branch and no skip path ever assigned to it. Added a workflows_skipped reason set by all three skip branches (OAuth stub missing, image build failed, server not ready), threaded back through run_pg/run_oci — both now return (exit_code, reason) — and acted on in main():

• clean run, workflows ran → 0, unchanged
• workflows skipped, everything that ran passed → non-zero, with the reason logged: "Workflow tests were SKIPPED (...) — the end-to-end API suite did not run, so this is NOT a pass. Everything that did run passed."
• a genuine failure → still returns its own exit code; a skip alongside it is reported as well rather than replacing it

Covered by six new tests in scripts/lib/tests/test_run_integration_tests.py (make test-dev-scripts, now 143 tests), including the exact shape reported here: go test exit 0, zero failed tests, zero failed packages, workflows never ran.

The .dockerignore cause of the underlying skip was already addressed separately, as noted in the issue.

[ericfitz]

Fixed by 41abfe54 — scripts/run-integration-tests.py now fails the run when the workflow suite is silently skipped, rather than exiting 0. Guarded by scripts/lib/tests/test_run_integration_tests.py.

68894272 added a second layer: build failures in the nested module are now named rather than swallowed, which is how the skip went unnoticed (#607).