Add multi-SKIP case to gfi-sweep eval suite - #1062
Conversation
Pattern 8's eval-suite layout omitted assertions.json, so a reader following it exactly produces cases with has_*/mention_* expected keys that silently fall back to MANUAL grading instead of PASS/FAIL. - Add assertions.json to the fixtures directory-layout block. - Add a paragraph explaining the has_*/mention_* structural-key rule and linking to eval-driven-development.md for the format. Closes apache#1004
Every existing step-3-present-proposals case has 0 or 1 SKIP issues, so skip_count_shown_as_summary_only was vacuously satisfied and a model that always returned true would pass. case-5-multi-skip adds 1 READY issue plus 6 SKIP issues spread across all three skip_reason categories (2 security-sensitive, 2 architectural-decision, 2 deprecation-decision) so the summary-only rule and the (security: M, architectural: K, deprecation: J) breakdown format actually get exercised. Closes apache#1001
…se (#1072) `case-2-mixed` already asserted `skip_count_shown_as_summary_only`, but with a single SKIP, where summarising and enumerating are nearly indistinguishable in the output — a model that lists skip titles individually could still pass. Add `case-5-multi-skip`: 2 READY + 1 NEAR-MISS + 4 SKIP spanning all three skip reasons (two security-sensitive, one architectural-decision, one deprecation-decision). Enumerating is now visibly wrong, so the summary-only rule is genuinely exercised, and the case checks it alongside `near_miss_has_label_proposal: false` rather than in isolation. It is also the first fixture in the suite to exercise the deprecation-decision reason. Supersedes #1062, which proposed the same case at the same paths with a different mix.
|
Superseded by #1072, merged as d0571f6. Both PRs added I went with #1072 because it is the later PR, its CI was green, and keeping a Your design had a real argument in its favour that I want on the record: Closing as superseded, not rejected. Thanks for the second look at the same |
2 participants
Summary
step-3-present-proposalscase in the good-first-issue-sweep eval suite has 0 or 1 SKIP issues, soskip_count_shown_as_summary_onlywas only ever vacuously satisfied — a model that always returnstruewould pass.case-5-multi-skip: 1 READY issue plus 6 SKIP issues spread across all threeskip_reasoncategories (2 security-sensitive, 2 architectural-decision, 2 deprecation-decision), so the summary-only rule and the(security: M, architectural: K, deprecation: J)breakdown format from the skill's Step 3 SKIP section actually get exercised.Closes #1001
Test plan
PYTHONPATH=tools/skill-evals/src python3 -m skill_evals.runner tools/skill-evals/evals/good-first-issue-sweep/step-3-present-proposals/fixtures/— all 5 cases load and render cleanly (exit 0).expected.jsoncontains all 8 output-spec fields with correct values for the new case.