ci: gate openjd-cli on the OpenJD conformance suite

[leongdl]

What changed

Two things, both about the conformance suite:

1. A Conformance workflow running the
   openjd-specifications
   suite against the installed openjd command on Linux, macOS and Windows, plus a
   DEVELOPMENT.md section on running it locally.
2. Two tests pinning the behaviour that
   specifications#174
   depends on.

Current result: 1160 passed, 0 failed on 2023-09/*.

Why the CI job

openjd-rs has had a conformance job for a while. This repo — the reference
implementation the suite's runner was written against — has had none, so a specification
regression here is caught only by whoever happens to run the suite by hand.

Why the tests

specifications#174 makes 187 conformance fixtures assert their own output: the action
runs the case's command as a child, echoes its output, and exits non-zero on mismatch.
Those verdicts only reach a runner if we do two things, neither of which had a test:

behaviour	test
A grandchild's output reaches our captured output	test_grandchild_output_captured_and_assertion_passes
A non-zero action exit fails the run	test_assertion_failure_fails_the_run

Regressing either would not fail this repo's suite. It would quietly weaken 187
conformance cases into "the task ran" — which is the false-pass shape that PR exists to
remove. The fixture takes a job parameter so the second test drives a real output
mismatch rather than asserting on a contrived failure.

No behavioural change; both pass against mainline as-is.

One wrinkle in the workflow

The job fixtures use command: python, and a bare Ubuntu runner provides only
python3. The workflow installs into a virtualenv outside the checkout and puts its bin
directory on PATH, which supplies python, python3 and the openjd entry point in
one step. Verified by simulating the workflow's steps locally, not just by reading the
YAML.

Testing

	result
2023-09/* against this CLI	1160 passed, 0 failed
Workflow steps simulated locally	python and openjd both resolve from the venv; suite passes
hatch run fmt	clean
hatch run lint	ruff, black, mypy all clean
hatch run test	309 passed, 2 skipped, coverage 92.98%

[github-actions]

No permissions: block, so this job inherits the repository/org default GITHUB_TOKEN scopes rather than least privilege. Every other workflow added to this repo declares them explicitly (permissions: {} in claude_pr_review_collect.yml, contents: read in release_publish.yml, security-events: write in codeql.yml).

This job only needs to read code, and it goes on to execute a test runner from another repository (see the openjd-specifications checkout below), so leaving the token broader than necessary is worth avoiding:

permissions:
  contents: read

Declaring it at the top level (or on the conformance job) keeps it consistent with the rest of .github/workflows/.

[github-actions]

This checks out openjd-specifications at whatever its default branch happens to point at, and the next steps execute code from it (uv run run_openjd_cli_tests.py, plus whatever dependencies uv resolves for that script). Two consequences:

1. Reproducibility — a push to openjd-specifications mainline can turn this PR check red without anything changing in this repo, and a previously-green commit will not necessarily re-run green.
2. Trust — the job runs arbitrary code from a floating ref of another repo. That is a sibling repo in the same org so the risk is modest, but combined with the inherited GITHUB_TOKEN scopes (see the earlier comment) it is worth constraining.

Consider pinning to a tag or commit SHA and bumping it deliberately, e.g.

      - name: Checkout openjd-specifications
        uses: actions/checkout@v7
        with:
          repository: OpenJobDescription/openjd-specifications
          ref: <tag-or-sha>
          path: openjd-specifications
          persist-credentials: false

persist-credentials: false is also worth adding here since nothing in the job pushes to that checkout.

[github-actions]

The workflow_call trigger and its tag / branch inputs are never exercised — no workflow in the repo calls ./.github/workflows/conformance.yml. Compare code_quality.yml, which declares the same input surface and is invoked from release_publish.yml:33 as the UnitTests job.

So the ref: ${{ inputs.tag }} plumbing below is currently dead: on pull_request the expression evaluates to the empty string and actions/checkout falls back to the event ref (correct behaviour, just not what the input is for). Either wire conformance into the release pipeline alongside UnitTests, or drop the workflow_call block and the ref: until it is needed — otherwise it reads as configured-and-working when it has never run.

Separately, branch is declared but unused even in the intended design (code_quality.yml has the same vestigial input, so this may just be copied).

[github-actions]

actions/checkout defaults to persist-credentials: true, which leaves the GITHUB_TOKEN in .git/config of this checkout for the life of the job. The last step then executes code from a different repository (uv run run_openjd_cli_tests.py) in the same workspace, and that suite in turn launches job fixtures that run shell/python commands. Any of that code can read $GITHUB_WORKSPACE/.git/config and use the token.

Nothing in this job pushes, so the credential is not needed:

      - name: Checkout
        uses: actions/checkout@v7
        with:
          ref: ${{ inputs.tag }}
          persist-credentials: false

This matters more here than in a typical build job precisely because the job’s purpose is to run externally-authored fixtures — and it compounds with the inherited token scopes noted above.

[github-actions]

Operational note: this job has no timeout-minutes and the workflow has no concurrency group. It runs the full conformance suite across three OSes, and the suite launches job fixtures that spawn subprocesses — a fixture that hangs will hold a runner for the default 6-hour limit, ×3.

concurrency:
  group: conformance-${{ github.ref }}
  cancel-in-progress: true

jobs:
  conformance:
    timeout-minutes: 30   # tune to observed runtime

The concurrency block also stops rapid pushes to a PR from stacking up three-OS runs; claude_pr_review.yml already uses this pattern in the repo.

[github-actions]

capture_output=True means the grandchild never writes to the action’s inherited stdout/stderr at all — the pipes are drained into completed.stdout/completed.stderr and the action then re-emits them through its own streams on lines 33-34.

So the runner only ever sees output produced directly by the process it spawned. If the CLI/session dropped grandchild output entirely, both new tests would still pass, because the bytes reach the log via the action’s own sys.stdout.write.

That undercuts the stated purpose. The template description says "losing grandchild output [...] silently weakens that whole suite", and test_self_asserting_task.py:17 is named test_grandchild_output_captured_and_assertion_passes with the comment "Missing means output from a process we did not spawn ourselves was dropped" — but nothing here exercises that path.

To actually pin grandchild output propagation the child has to inherit the streams rather than have them captured, e.g. print a second marker without capturing:

subprocess.run([sys.executable, "-c", "print(\"INHERITED:{{Param.Printed}}\")"])

and assert INHERITED:EXPECTED_VALUE shows up. As written, the fixture is a valid test of action exit-status propagation (which the second test does cover), just not of grandchild output capture.

[github-actions]

The comment claims "A virtualenv provides both names" (python and python3), but that is not true on Windows: CPython does not create a python3.exe in Scripts/ — only python.exe and pythonw.exe. A symlinked python3 exists only in POSIX venvs (bin/python, bin/python3, bin/pythonX.Y).

Not a bug for the stated need — the fixtures run command: python, which the Windows venv does provide — but if any fixture ever uses python3 the Windows leg will fail while Linux/macOS pass, and this comment would send the reader looking in the wrong place. Worth narrowing the claim to what is actually relied on (the venv provides python plus the openjd entry point) rather than asserting it covers all three names on every OS.

[github-actions]

OPENJD_CONFORMANCE_ASSERT_FAILED does not appear anywhere in this repo, and the fixture added in this same PR emits a different string — ASSERT_FAILED: expected output line not found (test/openjd/cli/templates/self_asserting_task.yaml:39), which is what the new tests grep for.

If the external suite really does emit OPENJD_CONFORMANCE_ASSERT_FAILED, this is fine but worth double-checking against the current openjd-specifications fixtures, since a reader following this doc will grep the log for a string that may not be there. If it was extrapolated from the local fixture, the doc and the fixture should use the same marker so the instruction is actually actionable.

[github-actions]

This negative assertion can never fire. The fixture writes the marker to stderr (sys.stderr.write("ASSERT_FAILED: ..."), self_asserting_task.yaml:39), but this checks outerr.out only. Note the sibling test on line 40 correctly checks outerr.out + outerr.err.

It is also largely redundant with expected_exit_code=0: the fixture only emits the marker on the same branch where it exits 1, so a run that reaches this line already cannot have failed the assertion. Either check both streams for symmetry with the other test, or drop the line.