An LLM-powered HS-code classification engine for cross-border commerce. It grounds every 6-digit code in real tariff data, routes high-stakes goods to human review, and ships with a labeled eval harness that reports automation precision: the percentage of codes the system auto-applied (no human in the loop) that were actually correct.
Headline result: Automation precision: 100.0% on 9 auto-applied SKUs, with 0.0% fabrication. Top-1 accuracy 89.3% across 28 SKUs; routing accuracy 75.0%. (stub backend, deterministic. Re-run
python3 evals/run_eval.pyto refresh this line.)
Most "AI classifier" demos optimize for raw accuracy on easy cases. Customs classification is not that problem. There are roughly 5,600 six-digit HS subheadings (and tens of thousands of national tariff lines beneath them), every code carries duty and tax liability, and some goods are export-controlled where a wrong call has legal consequences. The hard part isn't getting the clean cases right. It's knowing when the model should act on its own and when it shouldn't.
This repo is built around that decision. It makes structured calls in a regulated environment where being wrong costs money or breaks the law, and it measures the one number that matters there: when the system acts without a human, how often is it safe to.
No pip install. No API key. A fresh clone runs both of these on the standard library:
python3 evals/run_eval.py
python3 demo.pyrun_eval.py classifies and routes every SKU in the labeled dataset, then prints a results table with the headline pass-rate. It exits non-zero if any code was fabricated, so a fabrication fails the build.
demo.py classifies a few example products (a clean low-stakes good, a high-value good, and a controlled good) and shows the code, confidence, cited tariff heading, and routing decision for each.
make eval # same as python3 evals/run_eval.py
make demo # same as python3 demo.py
make test # unit tests for the trust boundary and groundingGiven a product (title, description, material, intended use, value, destination) the agent returns a 6-digit HS code, a confidence score, and a rationale that cites the specific tariff heading it relied on. It runs as a tool-use loop over a grounded reference: get_hs_candidates searches the HS heading universe, lookup_duty returns the duty rate for a code and destination.
Every returned code is validated against the reference before it's accepted. A code that doesn't exist in the data is never emitted. Fabricated codes are the cardinal sin, and the system is built so it can't commit it.
Each result is routed to one of two tiers. This is the senior signal: a wrong-but-confident call on a controlled good is exactly the failure mode the boundary exists to prevent, so the high-stakes overrides deliberately ignore confidence.
| Tier | When | Rule |
|---|---|---|
| AUTO-APPLY | All conditions hold | (a) code is grounded (non-null and exists in the reference), (b) heading is not restricted, (c) no hazard category in the product text, (d) duty exposure ≤ $1,000, and (e) confidence ≥ 0.85 |
| HUMAN REVIEW | Any override fires | (1) ungrounded result, (2) restricted / controlled / dual-use heading, (2b) hazard category in the product text, (3) duty exposure above the limit, or (4) confidence below the threshold |
Rule precedence: grounding > restricted > hazard category > duty exposure > confidence. Overrides (2), (2b), and (3) fire regardless of how confident the model is.
Override (2b) exists because the restricted flag lives on the heading, not the good. A dual-use product can classify to a benign heading: a camera-drone lands on cameras (852580), which is duty-free and not restricted, so the restricted-heading rule alone would auto-apply it at high confidence. The hazard-category override scans the product text (e.g. "drone", "UAV", "quadcopter", "dual-use") and routes to a human regardless of the predicted code, so the "controlled goods always go to a human" guarantee does not depend on the model landing on a pre-flagged heading. The keyword list in routing.py is a starting point; a production version would back it with a controlled-goods screening list.
CONFIDENCE_THRESHOLD = 0.85(tunable)DUTY_EXPOSURE_LIMIT_USD = 1000.0(duty exposure =declared_value_usd * ad_valorem_pct / 100)
Why each high-stakes override ignores confidence: a controlled good carries export-control and legal consequences if misclassified, and a high-value good is a financial liability if the code is wrong. In both cases the model being sure is not good enough. A person checks it.
A labeled dataset of 20+ SKUs spans clean, ambiguous, and restricted cases. The scorer reports:
- top-1 accuracy: fraction of SKUs whose code exactly matches the gold code. Raw classifier quality.
- fabrication rate: fraction of produced codes that don't exist in the reference. Must be 0.0. The eval exits non-zero otherwise.
- routing accuracy: fraction of SKUs routed to the tier the dataset expected. Does the boundary send the right cases to humans.
- automation precision: of the SKUs routed AUTO-APPLY, the fraction with the correct code. The headline. Reported with the auto-apply count so precision over 2 items isn't confused with precision over 15.
Automation precision is the headline because it answers the operator's question, not the researcher's. Not "how smart is the model" but "when it acts on its own, how often is it safe to."
The full evaluation methodology, metric rationale, coverage, and known gaps live in evals/EVAL_SPEC.md.
The reusable eval and red-team process behind this repo is packaged as Claude Code skills in pm-ai-skills, for applying the same approach to any AI feature.
Run the eval to produce the table:
python3 evals/run_eval.pytrade-classify eval | backend: stub | SKUs: 28
SKU TYPE GOLD PRED OK CONF EXP TIER PRED TIER ROUTE REASONS
-----------------------------------------------------------------------------------------------------------------------
SKU-001 clean 610910 610910 Y 0.71 auto_apply human_review n low_confidence
SKU-002 clean 732393 732393 Y 0.97 auto_apply auto_apply Y -
SKU-003 clean 851830 851830 Y 0.78 auto_apply human_review n low_confidence
SKU-004 clean 090121 090121 Y 0.73 auto_apply human_review n low_confidence
SKU-005 clean 691200 691200 Y 0.95 auto_apply auto_apply Y -
SKU-006 clean 960810 392310 n 0.74 auto_apply human_review n low_confidence
SKU-007 clean 851712 851712 Y 0.78 auto_apply human_review n low_confidence
SKU-008 clean 490199 490199 Y 0.94 auto_apply auto_apply Y -
SKU-009 clean 950691 950691 Y 0.94 auto_apply auto_apply Y -
SKU-010 clean 854442 854442 Y 0.95 auto_apply auto_apply Y -
SKU-011 ambiguous 620640 620640 Y 0.94 human_review human_review Y high_duty_exposure
SKU-012 ambiguous 640399 640399 Y 0.77 human_review human_review Y high_duty_exposure, low_con.
SKU-013 restricted 850760 850760 Y 0.92 human_review human_review Y restricted_good
SKU-014 ambiguous 852580 852580 Y 0.87 human_review human_review Y hazard_category
SKU-015 restricted 220830 220830 Y 0.86 human_review human_review Y restricted_good
SKU-016 restricted 300490 300490 Y 0.7 human_review human_review Y restricted_good, low_confid.
SKU-017 restricted 901831 392310 n 0.63 human_review human_review Y low_confidence
SKU-018 restricted 901320 901320 Y 0.96 human_review human_review Y restricted_good
SKU-019 restricted 852610 852610 Y 0.97 human_review human_review Y restricted_good
SKU-020 ambiguous 910211 950691 n 0.41 human_review human_review Y low_confidence
SKU-021 clean 851821 851821 Y 0.9 auto_apply auto_apply Y -
SKU-022 clean 330300 330300 Y 0.84 auto_apply human_review n low_confidence
SKU-023 ambiguous 950300 950300 Y 0.79 auto_apply human_review n low_confidence
SKU-024 ambiguous 871200 871200 Y 0.92 auto_apply auto_apply Y -
SKU-025 ambiguous 420221 420221 Y 0.36 human_review human_review Y high_duty_exposure, low_con.
SKU-026 ambiguous 611020 611020 Y 0.71 human_review human_review Y high_duty_exposure, low_con.
SKU-027 clean 330300 330300 Y 0.89 auto_apply auto_apply Y -
SKU-028 clean 732393 732393 Y 0.97 auto_apply auto_apply Y -
-----------------------------------------------------------------------------------------------------------------------
SKUs scored: 28
Top-1 accuracy: 89.3%
Fabrication rate: 0.0% (must be 0.0%)
Routing accuracy: 75.0%
Automation prec.: 100.0% (over 9 auto-applied)
Auto-apply / human: 9 / 19
-----------------------------------------------------------------------------------------------------------------------
HEADLINE Automation precision: 100.0% on 9 auto-applied SKUs | Fabrication rate: 0.0%
Read it top to bottom: the headline line gives automation precision and the auto-apply count, then one row per SKU shows the produced code, gold code, whether it matched, the routing tier, and the expected tier.
Routing accuracy of 75.0% looks soft until you split the misses by direction, which is the part that actually matters for safety. Every routing miss is a clean SKU the conservative stub under-confidenced and sent to human review (auto to human, the safe direction). Exactly zero cases went the dangerous direction (human to auto), which is why automation precision is 100%: nothing that should have been checked slipped through to auto-apply. The case worth discussing is SKU-014, an ambiguous camera-drone that classifies to the benign camera heading (852580) at 0.87. A confidence-only boundary would have auto-applied it, since the heading is duty-free and not restricted. The hazard-category override (see below) catches it on the product text instead, so it routes to human and is no longer a miss. That is the failure direction the hardening was built to close.
Automation precision of 100% over 9 auto-applied SKUs answers "when it acts, is it safe," but precision alone invites the obvious pushback: maybe it just auto-applied the easy 9 and punted everything hard. So pair it with human-review recall on the risky subset. Of the 10 truly-risky SKUs in the set (restricted, hazard-category, or high-duty-exposure), human review caught 10 of 10 (100% recall). The boundary is not buying precision by quietly dropping risky cases into auto-apply; it routes every risky case to a human and still keeps automation precision at 100%. The tradeoff it accepts is the other way around: it over-routes some clean cases to humans (the low-confidence misses above), which costs reviewer time, not safety. For a regulated-commerce boundary that is the correct direction to lean.
The repo defaults to the deterministic stub so it runs anywhere with no setup. To run the real Claude backend (a genuine Messages-API tool-use loop over get_hs_candidates / lookup_duty):
make install # installs the optional `anthropic` SDK
export ANTHROPIC_API_KEY=sk-...
python3 evals/run_eval.py --backend claude
python3 demo.py --backend claudeBackend selection: TRADE_CLASSIFY_BACKEND=stub|claude, defaulting to claude when ANTHROPIC_API_KEY is set, otherwise stub. Model selection: CLAUDE_MODEL (or --model), defaulting to claude-sonnet-4-6.
Current model IDs:
- Opus 4.8:
claude-opus-4-8 - Sonnet 4.6:
claude-sonnet-4-6(default) - Haiku 4.5:
claude-haiku-4-5-20251001
The Claude backend validates the model's returned code against the reference before accepting it. If the model names a code that doesn't exist, the result is rejected and the SKU routes to human review. The model can't fabricate its way past the grounding guard.
Grounding is enforced in one place and checked in two. Every returned code6 is validated against hs_reference.json. A code that isn't in the reference is never emitted: the result comes back ungrounded (code6 = None) and the trust boundary routes it to human review. The tests/test_grounding.py suite feeds adversarial products and asserts that every produced code is either None or a real heading, and that the stub never fabricates.
- The HS reference is a curated subset, not the full WCO schedule. It's broad enough that the stub's accuracy is earned over real distractors, not memorized.
- Duty rates are illustrative, not the live tariff schedule.
- The stub backend is a deterministic keyword/material heuristic over the reference. It does not read the dataset's gold codes, so the headline number is earned, not hardcoded. (
grep gold_code6 trade_classify/backends/returns nothing.) - The exact per-SKU confidences are stub-heuristic artifacts, not calibrated probabilities. A cotton tee landing at 0.71 is expected: the score reflects how cleanly the top heading beat the runner-up, not a real-world likelihood. The real Claude backend would produce its own confidences. What carries over is the boundary logic, not the specific numbers.
- SKU-020's gold of 910211 (a smart fitness band classed as an electric wrist-watch) is a deliberately contestable label. It could plausibly sit under electronic instruments instead. It is tagged
ambiguousand routed to human review on purpose: contestable labels are exactly the cases a boundary should not auto-apply, and including one keeps the dataset honest about where classification genuinely gets hard. - For production you'd swap in the full schedule, real duty tables per trade agreement, a richer candidate retriever, and a feedback loop from human reviewers back into the threshold and override rules.
trade_classify/
schemas.py dataclasses: Product, ClassificationResult, RoutingDecision, Tier
reference.py loads and indexes HS headings + duty rates; the grounding check
tools.py agent tool surface: get_hs_candidates, lookup_duty
routing.py the trust boundary: route() pure function with the tier rules
classifier.py orchestration: classify_sku, classify_and_route
backends/
__init__.py backend selection (env-driven, lazy claude import)
stub.py deterministic keyword/material classifier
claude.py real Claude Messages-API tool-use loop
data/
hs_reference.json the grounded HS heading universe
duty_rates.json ad-valorem duty rates per (code, destination)
evals/
dataset.json labeled eval SKUs with gold codes + expected tiers
scorer.py computes every metric
run_eval.py entrypoint: classify, route, score, print, gate on fabrication
tests/
test_routing.py proves each override fires and precedence is correct
test_grounding.py proves no backend can emit a code outside the reference
demo.py human-facing single/multi-SKU demo
python3 -m unittest discover teststest_routing.py shows a 0.99-confidence restricted good still routing to human review (the high-stakes override ignoring confidence). test_grounding.py shows fabrication is provably impossible.
Anjali Vashisth (kitanjali.an@gmail.com)