Skip to content

docs: add skill cards for the eight kermt-* agent skills - #26

Open
trvachov wants to merge 1 commit into
mainfrom
chore/skill-cards
Open

docs: add skill cards for the eight kermt-* agent skills#26
trvachov wants to merge 1 commit into
mainfrom
chore/skill-cards

Conversation

@trvachov

@trvachov trvachov commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Adds a skill-card.md alongside each of the eight kermt-* skills, following the NVIDIA skill card format used across the NVIDIA/skills catalog (326/326 skills there carry one).

Why

The NVIDIA/skills sync workflow drops any skill missing skill-card.md, skill.oms.sig, or an eval dataset — the enforcement step removes the skill dir before the PR is created, so non-compliant skills never merge into the catalog. The same requirement is tracked as SRC-11 in bionemo-agent-toolkit's CONTRIBUTING.

State of this repo against those three artifacts:

Artifact Status
eval dataset present — 4–5 tasks per skill under evals/
skill-card.md this PR
skill.oms.sig outstanding — needs nvskills-ci wired here (see below)

What's in the cards

Per skill: description, owner, license, use case, requirements/dependencies (Docker, NVIDIA Container Toolkit, GPU, optional WANDB_API_KEY), risks and mitigations, references, output types, evaluation, and ethical considerations.

Risk sections are specific rather than generic — GPU-hour cost on the detached training skills, catastrophic forgetting on kermt-continue-pretrain, the randomly-initialized decoder on kermt-add-cmim-pretrain, checkpoint-type validation on kermt-infer, and third-party data transmission when W&B tracking is enabled.

Evaluation results are marked pending, not fabricated. NVSkills-Eval has not been run against these skills; the cards state the eval task counts that exist and say results and a BENCHMARK.md will follow.

Still needed after this: signatures

skill.oms.sig cannot be added by a PR — it is emitted by the NVIDIA/nvskills-ci service. Wiring it here needs:

  1. a request-nvskills-ci.yml workflow (this repo currently has no .github/workflows/ at all),
  2. the NVSKILLS_CI_DISPATCH_TOKEN secret,
  3. a maintainer commenting /nvskills-ci on a PR.

Happy to open that as a follow-up if it's wanted here.

Context

bionemo-agent-toolkit vendors these eight skills via components.d/kermt.yml and rsyncs with --delete, so a card committed there is wiped on the next nightly sync. This repo is the source of truth. The catalog carries an interim copy of these same cards in a compliance.d/ overlay, which is retired automatically once this merges.

Please correct anything I got wrong about the skills' behavior or risks — I wrote these from the SKILL.md files and repo layout, not from running the pipelines.

🤖 Generated with Claude Code

Adds a skill-card.md alongside each skill, following the NVIDIA skill card
format used across the NVIDIA/skills catalog (description, owner, license, use
case, requirements, risks and mitigations, references, outputs, evaluation, and
ethical considerations).

These are required for catalog admission: the NVIDIA/skills sync drops any skill
missing skill-card.md, skill.oms.sig, or an eval dataset. Evals are already
present here; cards close the second gap. Signatures remain outstanding and need
nvskills-ci wired in this repo.

Evaluation results are marked pending rather than fabricated — NVSkills-Eval has
not yet been run against these skills.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Timur Rvachov <trvachov@nvidia.com>
@greptile-apps

greptile-apps Bot commented Aug 4, 2026

Copy link
Copy Markdown

Greptile Summary

Adds NVIDIA-format skill cards for all eight KERMT agent skills so they can satisfy catalog synchronization requirements.

  • Documents each skill’s purpose, prerequisites, outputs, risks, mitigations, evaluation coverage, and ethical considerations.
  • Marks evaluation results as pending and records the current source revision.
  • Two cards contain behavioral claims that should be corrected: monitor stale-manifest reconciliation and inference output row correspondence.

Confidence Score: 4/5

The documentation should be corrected before merging because it can mislead users about stale-run detection and positional alignment of inference results.

The monitor does not implement the advertised reconciliation of stale manifests with live containers, and inference preprocessing can remove invalid input rows despite the card promising one output row per input molecule.

Files Needing Attention: agent/skills/kermt-monitor/skill-card.md; agent/skills/kermt-infer/skill-card.md

Important Files Changed

Filename Overview
agent/skills/kermt-add-cmim-pretrain/skill-card.md Adds an accurate card for hybrid checkpoint conversion and continued pretraining; no actionable discrepancy was established.
agent/skills/kermt-continue-pretrain/skill-card.md Adds an accurate continued-pretraining card, including checkpoint-mode dispatch, detached execution, and evaluation coverage.
agent/skills/kermt-embed/skill-card.md Accurately documents encoder compatibility, per-readout arrays, preserved row alignment, and validity metadata.
agent/skills/kermt-finetune/skill-card.md Adds a finetuning card whose checkpoint validation, split handling, outputs, and evaluation claims match the workflow.
agent/skills/kermt-infer/skill-card.md Correctly documents checkpoint and feature preparation behavior, but incorrectly promises one output row per original input molecule when cleaning removes invalid SMILES.
agent/skills/kermt-monitor/skill-card.md Correctly covers monitoring prerequisites and outputs, but overstates the workflow’s ability to detect stale manifest-to-container mappings.
agent/skills/kermt-pretrain-scratch/skill-card.md Adds an accurate card for fresh-vocabulary, detached from-scratch pretraining and its associated resource risks.
agent/skills/kermt-setup/skill-card.md Accurately documents image reuse, conditional build behavior, and the containerized GPU smoke test.

Reviews (1): Last reviewed commit: "docs: add skill cards for the eight kerm..." | Re-trigger Greptile


## Known Risks and Mitigations: <br>
Risk: A stale or missing `run.json` can cause the skill to report on the wrong run, or to report nothing while a job is in fact still consuming GPU-hours. <br>
Mitigation: The skill cross-checks `run.json` against live Docker container state rather than trusting the manifest alone. <br>

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Stale manifests are not reconciled

When run.json is stale or lacks container_name, the monitor queries Docker using manifest-derived information or an mtime-based fallback rather than independently reconciling the run and container, causing missing or wrong-run status while the intended job can continue consuming GPU resources.

Suggested change
Mitigation: The skill cross-checks `run.json` against live Docker container state rather than trusting the manifest alone. <br>
Mitigation: The skill queries live Docker state using the container information available from `run.json`; when the manifest lacks a container name, users should provide `--container` to avoid relying on heuristic discovery. <br>

## Skill Output: <br>
**Output Type(s):** [Analysis, Files] <br>
**Output Format:** [CSV of per-molecule predictions; Markdown summary] <br>
**Output Parameters:** [2D — one row per input molecule, one column per predicted task] <br>

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Invalid rows break output alignment

When the input contains invalid SMILES, preprocessing removes those records before prediction despite the card promising one output row per input molecule, causing consumers that align predictions by original row position to associate results with the wrong molecules.

Suggested change
**Output Parameters:** [2D — one row per input molecule, one column per predicted task] <br>
**Output Parameters:** [2D — one row per valid input molecule after cleaning, one column per predicted task] <br>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant