Skip to content

docs: pretrained model zoo, weight-availability clarity & loud missing-weights warning - #457

Open
frgfm wants to merge 7 commits into
mainfrom
claude/vigilant-pare-213a22
Open

docs: pretrained model zoo, weight-availability clarity & loud missing-weights warning#457
frgfm wants to merge 7 commits into
mainfrom
claude/vigilant-pare-213a22

Conversation

@frgfm

@frgfm frgfm commented Jun 3, 2026

Copy link
Copy Markdown
Owner

Why

I went through every issue and discussion opened by non-maintainers and clustered the recurring friction. Almost all of it traces back to documentation blind spots around pretrained weights and loss usage:

Theme Reported in Root cause
"Why can't I load weights / which models are pretrained?" #123, #253, #230 No model zoo; weights advertised but availability never stated; detection has no weights
Weights ≠ torchvision #123 README claimed "maximum compatibility"; never said weights are Holocron-specific
Loss errors (dtype, ignore_index) #255, #211 poly_loss docstring said "predicted probability"; no usage example anywhere
Differs from reference impls #107 Design philosophy undocumented
Repeated "please share a repro / env" #255, #211 Blank issues were allowed, bypassing the (good) bug template

While verifying, I found the flagship Quick Tour example was broken: it called repvgg_a0(pretrained=True) then read model.default_cfg['input_shape'] — but default_cfg is empty for that model (KeyError) — and the snippet never imported torch. I also found that requesting weights for a model that has none silently returned a randomly-initialized model (only a logging call, usually invisible in notebooks).

What changed

Docs (P0)

  • Model zoo table in README.md (collapsible) and docs/docs/index.md — generated from the Checkpoint metadata: 29 classification + 1 segmentation pretrained checkpoints with training dataset, top-1 accuracy, and params.
  • Explicit note that weights are Holocron-trained (mostly Imagenette), not interchangeable with torchvision/timm, and that object detection ships no weights yet.
  • Softened the misleading "maximum compatibility" line.
  • blank_issues_enabled: false so the bug template (repro + collect_env) is enforced.

Loss docs (P1)

  • New "Loss functions" usage snippet in the README.
  • PolyLoss class docstring now documents shapes, the torch.int64 requirement, ignore_index, and a runnable example; fixed the poly_loss arg descriptions ("predicted probability" → raw logits / int64 class indices).

Behavior (P1)

  • load_pretrained_params() and _handle_legacy_pretrained() now raise an explicit UserWarning (in addition to the log) when no checkpoint is available, with an actionable message + the offending model class.

Quick Tour fix

  • Switched to darknet24 (actually ships weights and has a populated default_cfg) and added the missing import torch.

Implementation choices

  • Warn, don't raise. 47 of 56 entry points have no weights and the previous contract was graceful fallback, so raising would break the Quick Tour and any pretrained=True call. I kept the fallback but made it loud via the standard warnings system (so it shows in notebooks and can be escalated with python -W error::holocron.models.checkpoints.PretrainedWeightsUnavailableWarning). Easy to switch to a hard error later if you prefer.
  • Accuracy column added. When we scoped this I believed metrics weren't stored — they are, on the new Checkpoint API (evaluation.results), so I included top-1 accuracy at zero extra cost. Imagenette (10-cls) vs ImageNet-1k (1000-cls) is called out so the numbers aren't misread. Easy to drop if unwanted.
  • Two sources of truth reconciled. Availability is derived from both the new Checkpoint enums and the legacy default_cfgs['url'], so the table reflects exactly what pretrained=True loads (e.g. darknet24/tridentnet50 are legacy and have no stored accuracy → shown as "—").
  • No reformat / scope creep. Only the lines above changed; ruff/ty clean.

How to test

# from the repo root, with the dev env installed
pytest tests/test_models.py tests/test_nn_loss.py        # incl. new warning regression test
ruff check . && ruff format --check .
ty check                                                 # type check

# docs render (CI uses cairo for social cards; locally you can disable that plugin)
mkdocs build -f docs/mkdocs.yml

Manual checks:

import warnings, torch
from holocron.models.classification import darknet24, convnext_tiny

# 1) Quick Tour model now actually loads weights + has a usable default_cfg
m = darknet24(pretrained=True).eval()
assert {"input_shape", "mean", "std", "classes"} <= set(m.default_cfg)

# 2) missing weights now warns loudly instead of silently returning random init
with warnings.catch_warnings(record=True) as w:
    warnings.simplefilter("always")
    convnext_tiny(pretrained=True)
assert any(issubclass(x.category, UserWarning) for x in w)

# 3) PolyLoss usage from the docs runs as documented
from holocron.nn import PolyLoss
PolyLoss(ignore_index=-100)(torch.rand(4, 10, requires_grad=True), torch.tensor([0, 9, 3, 1])).backward()

Notes / possible follow-ups (not in this PR)

  • A short roadmap (e.g. YOLOv5/v7/DETR asks in Yolo v5/v7 #230) and a transfer-learning quickstart (Error Resnet pretrained #123) — deferred.
  • The README Latency benchmark table still lists a few models that have since gained/lost weights; could be regenerated separately.

🤖 Generated with Claude Code

@github-actions github-actions Bot added topic: docs Improvements or additions to documentation module: nn module: models ext: tests Related to test labels Jun 3, 2026
@codecov

codecov Bot commented Jun 3, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.89%. Comparing base (a642032) to head (6572b3f).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #457      +/-   ##
==========================================
+ Coverage   93.80%   93.89%   +0.08%     
==========================================
  Files          62       62              
  Lines        3794     3799       +5     
==========================================
+ Hits         3559     3567       +8     
+ Misses        235      232       -3     
Files with missing lines Coverage Δ
holocron/models/checkpoints.py 100.00% <100.00%> (+2.12%) ⬆️
holocron/models/classification/rexnet.py 100.00% <ø> (ø)
holocron/models/utils.py 94.87% <100.00%> (ø)
holocron/nn/functional.py 92.34% <100.00%> (+0.59%) ⬆️
holocron/nn/modules/loss.py 91.46% <ø> (+1.21%) ⬆️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@frgfm frgfm self-assigned this Jun 3, 2026
@frgfm

frgfm commented Jun 3, 2026

Copy link
Copy Markdown
Owner Author

Code Review — PR #457

Overall this is a well-scoped, high-value PR. The problem statement is precise, the Quick Tour fix is correct, and the loud missing-weights warning is a much-needed improvement. The docstring work on PolyLoss is thorough. A few implementation quality and maintainability items below.


1. Duplicated warning message — extract a helper

The same logger.warning + warnings.warn pattern is copy-pasted in two places with slightly different wording:

  • checkpoints.py:107-114"No pretrained weights are available for this model"
  • utils.py:108-114f"No pretrained weights are available for {model.__class__.__name__}"

Problem: When the message needs updating (e.g. adding a model name, changing the docs URL), you'll have to find and change both. The two paths also produce slightly different user-facing text for the same situation, which is confusing.

Suggestion: Extract a single _warn_no_pretrained(name: str) helper in checkpoints.py (or a shared _messages.py) that both call sites invoke:

# checkpoints.py
def _warn_no_pretrained(model_name: str | None = None) -> None:
    target = f" for {model_name}" if model_name else ""
    msg = (
        f"No pretrained weights are available{target}; it will be randomly initialized. "
        "Browse the model zoo (https://frgfm.github.io/holocron/) or train your own "
        "(https://github.com/frgfm/Holocron/tree/main/references)."
    )
    logger.warning(msg)
    warnings.warn(msg, UserWarning, stacklevel=3)

Then in _handle_legacy_pretrained you don't have the model class available, so call _warn_no_pretrained(). In load_pretrained_params you do, so call _warn_no_pretrained(model.__class__.__name__). Single source of truth, consistent text.


2. Model zoo table duplication — maintenance trap

The identical 29-row table is maintained in both README.md and docs/docs/index.md. Every time a model is added or accuracy changes, both files must be updated in lockstep. This will drift.

Suggestion (minimal): Add a comment at the top of each table in both files:

<!-- SYNC: this table must match the identical table in <other file> -->

Better long-term: Auto-generate the table from the Checkpoint metadata in a CI step or a small script. The data already lives in _checkpoint() calls — the table is derivable.


3. Test coverage gap — _handle_legacy_pretrained path untested

The new test test_load_pretrained_params_warns_without_url only covers load_pretrained_params. The _handle_legacy_pretrained path (used by ~50 model constructors) is not tested. A model constructor with pretrained=True and no checkpoint defined should also emit the warning.

Suggestion: Add a test that constructs a model via a legacy path (e.g. _handle_legacy_pretrained(True, None, None)) and asserts the UserWarning fires:

def test_handle_legacy_pretrained_warns_without_checkpoint():
    with pytest.warns(UserWarning, match="No pretrained weights"):
        _handle_legacy_pretrained(True, None, None)

This ensures both warning paths stay exercised as the codebase evolves.


4. stacklevel=2 correctness

In load_pretrained_params (called from _configure_model → model constructor → user code), stacklevel=2 points the warning to _configure_model, not the user's call site. This is acceptable but not ideal — the user sees a warning pointing at an internal function.

Options:

  • Acceptable as-is (the message is actionable regardless of where it points).
  • If you want the warning to point to the user's darknet53(pretrained=True) call, stacklevel would need to be dynamic based on call depth. This is fragile and not worth the complexity.

No action required — just noting for awareness.


5. poly_loss functional docstring incomplete

The PolyLoss class docstring is now excellent (shapes, dtype, ignore_index, runnable example). The poly_loss function docstring in functional.py:570-571 was updated to say "logits" and "int64", but it's missing:

  • The ignore_index parameter documentation (it's in the signature but not the docstring)
  • The TypeError / ValueError raises section that the class documents

Suggestion: Mirror the class's Raises section and add the ignore_index param doc to the function, or at minimum add ignore_index since users may call the functional API directly.


6. Minor nits

  • README.md — the darknet24 row shows for Top-1 accuracy. If accuracy data exists for darknet19 and darknet53 (same family), it's odd that darknet24 has none. Worth adding a brief note why (e.g. "legacy checkpoint, metrics not recorded") so users don't think it's an omission.
  • The blank_issues_enabled: false change is clean and correct.

Summary

Item Severity Effort
Extract shared warning helper Medium Low
Deduplicate model zoo table Low (now) / Medium (over time) Low
Test _handle_legacy_pretrained path Medium Low
poly_loss functional docstring gaps Low Low
darknet24 accuracy explanation Low Trivial

The PR is solid. The warning dedup and the missing test are the two items I'd address before merging. The rest are nice-to-haves that can ship as follow-ups.

@frgfm

frgfm commented Jun 3, 2026

Copy link
Copy Markdown
Owner Author

Review: PR #457

This is a well-structured PR with clear motivation, thorough investigation, and thoughtful implementation choices. The doc improvements address real user friction, and the warning behavior change is exactly the right level of intervention. A few areas worth tightening before landing:


1. Model zoo table is hardcoded in two places (maintenance debt)

The table appears in both README.md and docs/docs/index.md as hand-written markdown. The PR description says it was "generated from the Checkpoint metadata" but it is actually manually duplicated. This will drift — adding a new checkpoint variant requires updating both files and keeping them in sync.

Suggestion: Define the zoo data once (e.g., a dict or CSV in a script) and have a small generation step that writes both files, or at minimum add a comment at the top of each table pointing to the other copy so maintainers know.


2. Test coverage gaps

Only one test was added (test_load_pretrained_params_warns_without_url), and it tests load_pretrained_params(model, None) directly — which bypasses the model factory path that most users hit.

Missing tests:

  • Factory-path warning: convnext_tiny(pretrained=True) should warn. This exercises _handle_legacy_pretrainedwarnings.warn, not load_pretrained_params directly. These are two separate entry points and both should be covered.
  • No-false-positive guard: darknet24(pretrained=True) should not warn. Without this test, a future refactor that accidentally fires the warning on weight-bearing models won't be caught.
  • PolyLoss dtype enforcement: The docstring says TypeError is raised for non-int64 targets, but there is no test for this. This is the kind of contract that benefits from a regression test.

3. PolyLoss docstring example doesn't demonstrate ignore_index

The example sets ignore_index=-100 but uses target = torch.tensor([0, 9, 3, 1]) where no sample is masked, so the parameter is dead in the example. Since ignore_index is one of the friction points reported in issues #255/#211, showing it in action would be more instructive:

target = torch.tensor([0, -100, 3, 1])  # sample 1 is masked

4. *args/**kwargs in the PolyLoss docstring are opaque

The Args: section documents *args: args of Loss and **kwargs: keyword args of Loss. For documentation rendering and IDE autocomplete, listing the actual parameters (weight, ignore_index, reduction) by name would be clearer. Example:

Args:
    weight: class weight for loss computation (default: None)
    ignore_index: specifies target value that is ignored (default: -100)
    reduction: type of reduction to apply (default: "mean")
    eps: epsilon 1 from the paper (default: 2.0)

5. (Minor) Legacy models still bypass the checkpoint system

darknet24 and tridentnet50 use the old default_cfgs dict pattern (see darknet.py:135-138) instead of the new Checkpoint/_configure_model path. This is why they show for accuracy in the table. Not a regression — pre-existing — but if they were migrated to the Checkpoint system, the table could populate their accuracy automatically and the two sources of truth (default_cfgs vs Checkpoint) would be consolidated.


6. (Nit) stacklevel choice for warnings.warn

_handle_legacy_pretrained uses stacklevel=2, which points the warning to the model factory function (e.g., convnext_atto). That's reasonable. load_pretrained_params also uses stacklevel=2, which when called via _configure_model points to _configure_model rather than the user's code. If someone calls darknet24(pretrained=True), stacklevel=3 in _handle_legacy_pretrained would point to their invocation in user space, which is more actionable. Minor nit, worth a thought.


Verdict

The core changes are solid — clean, minimal, and well-targeted at real user problems. Address the duplication in (1) and the test gaps in (2) before shipping, and give (3)–(4) a quick pass.

@frgfm

frgfm commented Jun 3, 2026

Copy link
Copy Markdown
Owner Author

Review Summary

This PR effectively addresses critical documentation gaps that were causing user confusion around pretrained weights and loss functions. The changes are well-organized and solve real pain points identified in user issues.

Strengths

  1. Comprehensive Documentation: The model zoo table provides clear visibility into which models have pretrained weights, their accuracy, and parameter counts - exactly what users were asking for.

  2. Clear Communication About Weight Compatibility: Explicitly stating that weights are Holocron-trained (mostly Imagenette) and not interchangeable with torchvision/timm prevents a major source of confusion.

  3. Improved Developer Experience:

    • The explicit UserWarning when weights are missing is much better than silent fallback
    • Fixed Quick Tour example that actually works now
    • Clear loss function documentation with runnable examples
  4. Process Improvements: Disabling blank issues ensures better bug reports going forward.

Suggestions for Improvement

  1. Table Maintenance: While noting the table is generated from Checkpoint metadata is good, consider adding a comment in the documentation about how to regenerate it (or if it's automated) to help future maintainers.

  2. Warning Visibility: The UserWarning approach is good, but consider if we should also:

    • Add a link to the model zoo table directly in the warning message
    • Consider logging at WARNING level instead of just using warnings.warn() for better visibility in scripts
  3. Consistency Check: Verify that all 56 entry points mentioned actually behave consistently with the new warning approach. Spot-check a few more models to ensure they all show warnings when appropriate.

  4. Future-Proofing: The accuracy column is valuable, but consider if we should add a footnote about potential future migration to ImageNet-1k accuracies for certain models as they get updated.

Minor Nitpicks

  1. In the PolyLoss docstring example, the comment says - this is correct but could be strengthened by showing what happens if you use the wrong dtype (though that might be beyond scope).

  2. Consider adding a brief note in the README about how users can contribute to expanding the model zoo (training scripts, etc.) to encourage community involvement.

Conclusion

This is a strong PR that significantly improves the user experience. The changes are focused, well-documented, and address the root causes of user friction. With the minor suggestions above, it would be even more robust.

LGTM - looks good to merge!

@frgfm frgfm left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review

Solid PR — the cluster-by-issue approach is the right way to make a documentation/warning change, and the tests are well-targeted. That said, there are a few implementation-quality and maintainability issues that I'd like to flag before landing. The current state is fine for pretrained=True users but creates maintenance debt that will compound. Most are small; a few are real bugs.

Note on the failing CI checks (astral, headers, deps-sync, codecov): I checked and these are pre-existing on mainastral fails on holocron/trainer/core.py:155 (fastprogress typing, untouched here), headers flags 53 files whose copyright line still ends in -2025 (unrelated), and deps-sync complains about a pytest/Pillow mismatch between api/pyproject.toml and ./pyproject.toml. None of these are introduced by this PR. Not blocking from a review standpoint, but worth a separate fix-up issue.


Critical

1. Warning message is duplicated verbatim across two files — collapse to a single source of truth. holocron/models/utils.py:107-114 and holocron/models/checkpoints.py:106-112 carry the same multi-line msg = (...). block. Same wording, same URLs, same logger.warning + warnings.warn pair. If the model-zoo URL ever changes, you'll fix it in one and forget the other. Suggested path:

  • Move the message into a single helper (e.g. holocron/models/utils.py::warn_pretrained_unavailable(model_cls=None)) and call it from both sites. That also fixes the asymmetry where load_pretrained_params has f"... for {model.__class__.__name__}" but _handle_legacy_pretrained doesn't (the legacy path doesn't have the class handy — so the helper should take an optional model_cls and degrade gracefully).

2. stacklevel=2 is wrong for both warnings — should be stacklevel=3 in load_pretrained_params. When a user runs darknet24(pretrained=True), the call chain is darknet24 → _handle_legacy_pretrained/_configure_model → load_pretrained_params → warnings.warn. With stacklevel=2, the warning is attributed to load_pretrained_params itself, not the user's darknet24(pretrained=True) call site. That's exactly the source of confusion the warning is supposed to resolve. The test passes because it calls load_pretrained_params directly (only one frame above warn), so the bug is invisible to the test. The same issue applies in _handle_legacy_pretrained (called from model factories) and to the Quick Tour in the README.

3. The new warning path in checkpoints.py is untested. Codecov is right: 25% patch coverage on checkpoints.py because the new test only exercises load_pretrained_params. Add a parallel regression test in tests/test_models.py for _handle_legacy_pretrained — e.g. patch a checkpoint URL to None in ReXNet1_0x_Checkpoint.DEFAULT (or use a model whose default_cfg["url"] is None) and assert a UserWarning is raised with the offending class name. As written, the checkpoints.py change has no regression guard.

4. Each pretrained=True call now fires both logger.warning and warnings.warn with the same message. Two channels, one signal — Jupyter users will see it in the cell output, CLI users will see it in stderr and their log handler. Pick one. The warnings system is the right one here (visible in notebooks, escalatable with -W error), so I'd drop the logger.warning and let application code configure warning routing if it wants a log record.


Important

5. Use a dedicated warning subclass. A bare UserWarning is too generic — every project emits them, and users can't filter this specific class without string-matching the message. Something like:

class PretrainedWeightsUnavailableWarning(UserWarning):
    """Raised when pretrained=True is requested but no checkpoint is available."""

…in holocron/models/checkpoints.py, and emit that from the shared helper above. Users can then do warnings.filterwarnings("ignore", category=PretrainedWeightsUnavailableWarning) cleanly. The test can also assert on the class instead of a regex match (more robust to message edits).

6. The model zoo is hand-typed markdown and will go stale on the first new checkpoint. The PR description says "generated from the Checkpoint metadata" but the table is plain markdown in README.md (29 rows × 5 columns = 145 data points) and again in docs/docs/index.md (so 290 data points to keep in sync). Two ways out, in order of preference:

a. (best) Add a small references/generate_model_zoo.py (or a make target) that imports the registries, walks every Checkpoint enum, and writes the table to a <!-- AUTOGEN START --><!-- AUTOGEN END --> block in both files. CI runs the generator and fails on diff. ~50 lines.

b. (pragmatic) Use mkdocstrings snippets / markdown-include to pull the table from a single source file (e.g. docs/includes/model_zoo.md) and reference it from both README and index. Fewer LOC change but doesn't actually solve the "hand-typed" problem.

Either way, the hardcoded counts ("29 classification + 1 segmentation") at the bottom of the README and in the PR body are guaranteed to drift.

7. The PolyLoss example is now duplicated in four places. README (## Loss functions block), docs/docs/index.md (none added but README is mirrored), holocron/nn/modules/loss.py docstring, and tests/test_models.py style. If the constructor signature changes, four sites to update. The docstring is the natural single source — have the README import the same snippet via a mkdocstrings include or just paste the literal lines once and point at the class. At minimum, drop one.


Minor / Nitpicks

  • poly_loss functional docstring: the new target: description ("hard class indices … OR soft class probabilities …") duplicates shape info across the class/function docstrings. Consider: target: class indices of shape $(N, \\dots)$ with dtype ``torch.int64`` (use soft probabilities only via the class wrapper, which adds the log-softmax gate). Or just keep the class-level shape and have the function doc say "see PolyLoss for shape/dtype details".
  • The note block in the PolyLoss class docstring says "A TypeError is raised if the hard target is not of dtype torch.int64." — make it "is raised by F.poly_loss" to keep the wrapper docstring free of implementation-detail references.
  • PR body says warnings can be escalated with -W error — that's not a valid python -W value. The correct form is python -W error::UserWarning:holocron.models.utils your_script.py or PYTHONWARNINGS=error::UserWarning:holocron.models.utils. Tiny but people copy-paste from PR bodies.
  • stacklevel of 2 is the minimum useful value — and is what most call sites need, so it's defensible, but the helper from (1) should document this in its docstring so a future maintainer doesn't "fix" it to 1 or 2 again.
  • No way to test the "noisy in a loop" case. If a user calls darknet24(pretrained=True) in a tight training loop (e.g. unit tests, sweeps), the warning will dedupe naturally via __warningregistry__, but the test would be valuable to lock that behaviour in. Optional.
  • tests/test_models.py is already an oddly named file — it now contains a test for utils.load_pretrained_params, a test for utils.fuse_conv_bn, and a test for model_from_hf_hub. The new test fits the file's actual scope (models utilities), but if the file is renamed in the future, the new test will move with it — worth a one-line comment near the test pointing at _handle_legacy_pretrained so the corresponding test for the legacy path lives next to it.

Suggested follow-up issues (not blocking)

  • Migrate the 53 pre-existing files off the -2025 copyright header (or relax validate_headers.py to allow a 1-year grace period). Unrelated to this PR but it makes the headers check permanently red.
  • Fix the pre-existing astral failures in holocron/trainer/core.py (fastprogress typing).
  • Sync pytest and Pillow versions across api/pyproject.toml and ./pyproject.toml.

Happy to push a fix for items 1–5 myself if you'd like — they're small and mechanical. The rest are follow-ups.

@frgfm

frgfm commented Jun 3, 2026

Copy link
Copy Markdown
Owner Author

Review follow-up

I did another pass through PR #457. Direction is good: clearer pretrained-weight docs, a working quick-tour model, and a visible missing-weights warning are all useful. I would tighten these before merge.

  1. PolyLoss(ignore_index=-100) is now documented, but the implementation crashes before masking.

    In holocron/nn/functional.py:588, the hard-label path gathers with the raw target before ignored positions are sanitized. The new PolyLoss docs recommend the default ignore_index=-100, but this fails:

    import torch
    from holocron.nn import PolyLoss
    
    logits = torch.rand(4, 10, requires_grad=True)
    target = torch.tensor([0, -100, 3, 1])
    PolyLoss(ignore_index=-100)(logits, target)

    Result: RuntimeError: index -100 is out of bounds for dimension 0 with size 10.

    Suggested fix: compute the hard-target valid mask before gather, replace ignored target positions with a valid dummy class for the gather (for example target.masked_fill(~valid_idxs, 0)), then apply the existing reduction only over valid positions. Add a regression test using the default ignore_index=-100; otherwise the docs advertise a path that does not work.

  2. The new missing-weights warning has two entrypoints, but only one is tested.

    tests/test_models.py covers load_pretrained_params(model, None). The new warnings.warn path in holocron/models/checkpoints.py::_handle_legacy_pretrained is untested, which is also what Codecov flags.

    Minimal test:

    with pytest.warns(UserWarning, match="No pretrained weights"):
        _handle_legacy_pretrained(True, None, None)

    I would also add a no-warning guard for _handle_legacy_pretrained(True, None, default_checkpoint) so future refactors do not accidentally warn for models that do ship weights.

  3. Warning message construction is duplicated across checkpoints.py and utils.py.

    Both call sites build nearly the same long message and emit both logger.warning and warnings.warn. A small shared helper that accepts an optional model name would reduce code size, keep the docs/reference URLs in one place, and make stacklevel/category choices consistent. The helper can still preserve existing behavior if you want both logging and warnings.

  4. The model-zoo table is duplicated in README.md and docs/docs/index.md.

    The table matches the 29 checkpointed classification entries now, but maintaining 29 rows in two files will drift. Minimal path: put the table in one include/source file or mark generated blocks and add a tiny script/check that rewrites both copies from checkpoint metadata. That keeps the docs useful without turning them into another hand-maintained registry.

  5. Wording nit: "Top-1 accuracy is reported on each model's own training set" reads like the metric was measured on the training split. If these are validation metrics, say "validation split for the listed dataset" or similar.

Verification:

  • uv run --with pytest --with pytest-cov pytest tests/test_models.py::test_load_pretrained_params_warns_without_url -q passes.
  • The PolyLoss(ignore_index=-100) probe above fails on the PR head.

@github-actions github-actions Bot added func: build Related to build and installation ext: ci topic: style labels Jun 3, 2026
@frgfm

frgfm commented Jun 3, 2026

Copy link
Copy Markdown
Owner Author

Addressed the review feedback (3 commits)

Thanks for the thorough reviews — I catalogued every distinct point across all five, deduped, and prioritized by ROI.

🐛 The highest-ROI item turned out to be a real bug — and it's #211

While wiring up the loss docs I found that PolyLoss(ignore_index=-100) crashes (the follow-up review #1 nailed it): the hard-label path called gather with the raw target (−100 is out of bounds) and the mask guard ignore_index >= 0 skipped negative values entirely — so the conventional -100 never worked. My new docstring was actively recommending this broken path. Fixed in poly_loss: compute the ignore mask first, gather with a sanitized target, and exclude ignored positions for any ignore_index (loss and gradient zeroed, including under reduction="none"). Regression test uses the default -100. → 8ace4f3

Done

Feedback (reviewers) Change Commit
Warning duplicated; drop one channel; typed subclass; stacklevel (R1.1/1.2/1.4/1.5, R2.1, R3.6, R5.3) Single _warn_pretrained_unavailable() helper → one warnings.warn with a filterable PretrainedWeightsUnavailableWarning, stacklevel=3 (verified it now points at the model builder, e.g. convnext_tiny). Dropped the duplicate logger.warning and the now-dead logger. dbda380
Untested _handle_legacy_pretrained path + no-false-positive guard (R1.3, R2.3, R3.2, R5.2, codecov) Added both, plus ignore_index and dtype-TypeError tests. dbda380/8ace4f3
Model-zoo table hand-typed in two files → will drift (R1.6, R2.2, R3.1, R4.1, R5.4 — all 5) New .github/generate_model_zoo.py derives the table statically (stdlib AST, no torch) from the Checkpoint metadata + legacy default_cfgs, written between AUTOGEN markers. make model-zoo regenerates; make model-zoo-check is wired into pre-commit and a quality CI job (mirrors deps-check) and fails on drift. 196a25b
PolyLoss example doesn't exercise ignore_index; *args/**kwargs opaque (R3.3/3.4) Example now masks a sample with -100; real params listed. 8ace4f3
"training set" wording (R5.5); darknet24 needs explanation (R2.6) "validation split"; footnote added under the table. 196a25b

Correction to my PR description

-W error (R1 nit) isn't valid. With the new subclass: python -W error::holocron.models.checkpoints.PretrainedWeightsUnavailableWarning … or warnings.filterwarnings("error", category=PretrainedWeightsUnavailableWarning). (Body updated.)

Deferred — agree these are non-blocking follow-ups

  • Pre-existing red CI (headers year drift, astral fastprogress typing, deps-sync pytest/Pillow) — unrelated to this PR.
  • Migrating darknet24/tridentnet50/unet_rexnet13 off default_cfgs onto the Checkpoint API — would auto-fill their now- cells (the generator already prefers Checkpoint data when present).

Green locally: ruff, ty, pytest tests/test_models.py tests/test_nn_loss.py (15 passed), make model-zoo-check, mkdocs build.

🤖 Generated with Claude Code

frgfm and others added 7 commits June 26, 2026 19:22
…ability

Driven by recurring confusion in community issues/discussions (#107, #123, #211, #230, #253, #255):

- README + docs index: replace the inconsistent architecture list with an accurate model zoo table (29 classification + 1 segmentation checkpoints), generated from the Checkpoint metadata, with training dataset (Imagenette vs ImageNet-1k), top-1 accuracy and params.
- State clearly that weights are Holocron-trained and NOT interchangeable with torchvision/timm, and that object detection ships no pretrained weights yet.
- Fix the flagship Quick Tour: it called repvgg_a0 (whose legacy default_cfg is empty -> KeyError) and never imported torch. Now uses darknet24 and imports torch.
- Add a 'Loss functions' usage snippet and clarify PolyLoss/poly_loss docstrings: input is raw logits, hard targets must be torch.int64 class indices, ignore_index masks samples.
- Issue templates: disable blank issues so the bug template (repro snippet + env) is enforced.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… are missing

Previously, calling a builder with pretrained=True when no checkpoint URL is available logged a vague 'Invalid model URL' through the logging module (usually silent in notebooks/scripts) and returned a randomly-initialized model, so users could not tell their 'pretrained' model was actually random (cf. #123, #253).

load_pretrained_params() and _handle_legacy_pretrained() now also emit an explicit UserWarning with an actionable message pointing to the model zoo and reference scripts (load_pretrained_params additionally names the model class). Behavior is otherwise unchanged - it still falls back to random initialization - to stay backward-compatible. Adds a regression test.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
poly_loss crashed when the hard target contained ignore_index: gather() was called with the raw target (out of bounds for the default -100), and the mask guard 'ignore_index >= 0' skipped negative values entirely. This is the failure reported in #211, which the new docs were inadvertently recommending.

The hard-label path now computes the ignore mask first, gathers with a sanitized target, and excludes ignored positions for any ignore_index value -- zeroing their loss and gradient, including under reduction='none'. The PolyLoss docstring now documents the contract (logits in, int64 class indices), shows a runnable ignore_index example, and lists the real parameters. Adds regression tests for the default ignore_index=-100 and the int64 dtype guard.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… paths

The missing-weights notice was duplicated across checkpoints.py and utils.py (with slightly different wording) and emitted on two channels (logging + warnings). It now lives in a single _warn_pretrained_unavailable() helper that raises a dedicated, filterable PretrainedWeightsUnavailableWarning via warnings.warn only, with stacklevel=3 so the warning points at the model builder (e.g. repvgg_a0) instead of a Holocron internal.

Adds a regression test for the _handle_legacy_pretrained (model-factory) path and a no-false-positive guard for weight-bearing models, and relaxes the test lint rules (PLC2701) to allow importing the internal helper.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The model-zoo table was hand-maintained in both README.md and docs/index.md, guaranteeing drift on the next checkpoint. It is now generated by .github/generate_model_zoo.py, which derives the rows statically (stdlib only, no torch) from the Checkpoint metadata and the legacy default_cfgs, and writes them between AUTOGEN markers.

'make model-zoo' regenerates the tables; 'make model-zoo-check' (wired into pre-commit and a quality CI job, mirroring deps-check) fails on drift. Also corrects the accuracy wording (validation split, not training set) and demonstrates ignore_index in the README loss snippet.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…inks

Per review feedback:
- The Quick Tour (README + docs) uses repvgg_a0 again instead of darknet24 -- RepVGG is the more representative flagship. Since it uses the Checkpoint API, model.default_cfg is the Checkpoint object, so the snippet now reads model.default_cfg.pre_processing.{input_shape,mean,std} and .meta.categories (the original repvgg example was actually broken: it indexed default_cfg like a dict, which only works for the legacy models).
- Restored the per-task architecture lists with their paper links in the docs model zoo, alongside the pretrained-weights table. The links matter for SEO and for showing which architectures are implemented in a research repo. (The README's Paper references section was untouched.)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…t docstrings

Addresses two nits from the PR review:

- .github/generate_model_zoo.py: _checkpoint_entry now resolves the `DEFAULT = <member>` alias (and reads `arch` by keyword rather than position) instead of taking the first-declared checkpoint. The generated table is byte-identical today (every enum's DEFAULT happens to be its first member), but it no longer risks silently emitting the wrong row if a future enum defaults to a non-first member.
- rexnet.py: rexnet1_0x ships ImageNet-1k weights (not ImageNette) and rexnet2_2x ships Imagenette (not ImageNet); corrected both pretrained-arg docstrings to match their DEFAULT checkpoints (and the model-zoo table).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@frgfm
frgfm force-pushed the claude/vigilant-pare-213a22 branch from 3f6394a to 6572b3f Compare June 26, 2026 17:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ext: ci ext: tests Related to test func: build Related to build and installation module: models module: nn topic: docs Improvements or additions to documentation topic: style

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant