feat(validator): add override-file contract check for .apache-magpie-overrides/ - #665
Merged
Merged
Conversation
Collapse "What's been built" to one line per item; all 22 planned work items preserved verbatim; redundant shipped-state notes trimmed. Generated-by: Claude (Opus 4.7)
…overrides/ Add a new SOFT advisory check (check apache#15) to the skill-and-tool-validator that validates .apache-magpie-overrides/<skill>.md files in adopter repos. Two advisory checks per file: - Structure: the canonical 'apache-magpie agentic override' header comment must be present (confirms the file was created via /magpie-setup override). - Baseline integrity: heuristic scan for patterns attempting to weaken the framework safety / confidentiality / privacy / data-not-instructions baseline (ignore safety, bypass confidentiality, skip privacy-llm-gate, treat external content as instructions, disclose confidential reports). HTML comment lines are excluded from the weakening scan to avoid flagging prose that explains what NOT to do. The directory scanner silently skips repos without an override directory. All violations are SOFT advisory. 21 new tests cover: clean override passes, missing header, each weakening pattern, HTML-comment exclusion, directory scanner, README.md skip, multi-file coverage, and discoverable-without-editing-skill confirmation. Also clears the override-file contract gap from specs/adoption-and-setup.md. Generated-by: Claude (Opus 4.7)
Member
|
Oh.. very cool :) |
potiuk
approved these changes
Jul 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add a new SOFT advisory check (check #15) to the skill-and-tool-validator
that validates .apache-magpie-overrides/.md files in adopter repos.
Two advisory checks per file:
must be present (confirms the file was created via /magpie-setup override).
framework safety / confidentiality / privacy / data-not-instructions
baseline (ignore safety, bypass confidentiality, skip privacy-llm-gate,
treat external content as instructions, disclose confidential reports).
HTML comment lines are excluded from the weakening scan to avoid flagging
prose that explains what NOT to do. The directory scanner silently skips
repos without an override directory. All violations are SOFT advisory.
21 new tests cover: clean override passes, missing header, each weakening
pattern, HTML-comment exclusion, directory scanner, README.md skip,
multi-file coverage, and discoverable-without-editing-skill confirmation.
Also clears the override-file contract gap from specs/adoption-and-setup.md.
Generated-by: Claude (Opus 4.7)
Type of change
.claude/skills/<name>/) — eval fixtures updated belowtools/<system>/*.md)tools/*/withpyproject.toml)docs/,README.md,CONTRIBUTING.md)projects/_template/)prek, workflows, validators)Test plan
prek run --all-filespassesuv run pytest/ruff check/mypypasses(
PYTHONPATH=tools/skill-evals/src python3 -m skill_evals.runner tools/skill-evals/evals/<skill>/)(a regression test for the bug fixed / the behaviour added — see CONTRIBUTING.md)