Skip to content

ci_mode_weakened cannot fire through the real verify --base path #298

Description

@pengfei-threemoonslab

SHIP-VERIFY-POLICY-WEAKENED's ci_mode_weakened disjunct — the check that catches an agent downgrading the CI gate from strict to advisory — never fires in a real run.

Mechanism (traced 2026-07-28):

  1. cli/verify/orchestrator.py::_prepare_base_report scans the base tree with run_scan(..., ci_mode="advisory", ...).
  2. cli/scan/prepare.py:33 applies that CLI override by mutating the loaded manifest: manifest.ci.mode = ci_mode.
  3. cli/scan/final_report.py:68 builds the report's effective_policy snapshot from sanitized.manifest.ci.mode.

So every cached base report records effective_policy.ci_mode == "advisory" regardless of what the base manifest declared. checks/verify_policy.py::_compare compares that against the head manifest's real mode, so a PR that downgrades ci.mode from strict to advisory yields head_rank == base_rank and emits nothing.

Reproduction. Fresh git repo with an OpenAI Agents SDK tool → agents-shipgate init --write → commit with ci.mode: strict → edit the worktree manifest to ci.mode: advisoryagents-shipgate verify --base HEAD. The base cache report under .git/agents-shipgate/base-scans/*/report.json shows "ci_mode": "advisory", and no ci_mode_weakened finding appears in agents-shipgate-reports/report.json.

Why coverage missed it. tests/test_verify_weakening.py injects the base EffectivePolicy directly instead of driving a real base scan, so the unit tests pass while the end-to-end path is blind.

Scope to check. fail_on_loosened and severity_override_lowered need the same audit — _prepare_base_report also passes fail_on=None, which may blank effective_policy.fail_on the same way.

Likely fix. Build the effective-policy snapshot from the manifest as loaded from disk, before CLI overrides are applied (or record the pre-override mode alongside), so the snapshot describes the repository's declared policy rather than the invocation's. Add an end-to-end regression test that drives a real base scan.

Severity. A PR touching shipgate.yaml still routes to human review via SHIP-VERIFY-TRUST-ROOT-TOUCHED, so this is a missed specific finding rather than a silent merge — but the specific finding is the one that names the weakening, and the flagship claim is that an agent cannot quietly weaken its own gate.

Found while working on #297.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions