Enforce mandatory precision translation

.agents/skills/book-translator/SKILL.md

@@ -16,9 +16,12 @@ its existing OPDS browser.
 
 ```
 request → search (Gutenberg / Standard Ebooks / Anna's / local)
-        → fetch source text
-        → extract chapters
-        → GLM-5.2 paragraph-wise translation (batches, retries)
+        → autonomously research provenance and prefer a verified public-domain
+          original-language edition over a relay translation
+        → fetch source text → extract chapters
+        → build a web-grounded, independently cross-checked and normalized concept sheet
+        → GLM-5.2 precision translation + semantic proofread
+        → fail-closed whole-book terminology audit and contextual repair
         → assemble cp-original/cp-translation bilingual EPUB
         → calibredb add → Content Server OPDS feed
         → best-effort HTTP POST push to device (if File Transfer on)
@@ -33,11 +36,14 @@ long-presses Confirm to cycle Both → Original only → Translation only.
 
 - Pipeline code: `~/github/BabelForge/`
 - Config: `~/github/BabelForge/config.py`
-- State machine: `data/catalog.json` (requested → searching → selected →
-  translating → assembling → published / failed)
+- State machine: `data/catalog.json` (requested → researching → selected →
+  analyzing → translating → quality_checked → assembling → published / failed)
 - Calibre library: `~/Calibre-Library/` (served by launchd
   `com.local.calibre-server` on port 8080)
-- Public OPDS: `https://books.getlingo.store/opds` (cloudflared tunnel)
+- Public OPDS: `https://books.foundby.kr/opds` (cloudflared tunnel). X3/X4
+  setup uses `http://books.foundby.kr/opds`: the catalog is public/no-auth and
+  current reader firmware fails the Cloudflare TLS handshake while HTTP serves
+  the same feed without redirecting.
 
 ## How the agent drives it
 
@@ -47,8 +53,13 @@ tools instead of shelling out — no paths, no JSONL parsing:
 
 - `search_books(query)` → candidate list (prefix `annas:` to opt in; pass a file
   path for a local file). Show up to 8, let the user pick one.
-- `translate_book(candidate)` → returns a `req_id` immediately; the pipeline runs
-  in the background (~1-2h). Pass the exact candidate dict the user chose.
+- `translate_book(candidate)` → returns a `req_id` immediately; the precision
+  pipeline runs in the background. Pass the exact candidate dict the user chose;
+  it may autonomously replace a relay translation with a verified public-domain
+  original-language edition.
+- `retry_request(req_id)` → resumes a failed request under the same `req_id`,
+  reusing its verified source, concept sheet, and translation checkpoint. This
+  is the normal recovery path and does not require shell execution approval.
 - `get_status(req_id)` → progress %, then `published` (epub_path/opds_url) or
   `failed` (error). A failed run is resumable — re-running continues from its
   checkpoint.
@@ -92,20 +103,36 @@ cd ~/github/BabelForge
 `start.sh` runs `run_request.sh` detached (nohup); log lands in
 `logs/pipeline.<timestamp>.log`. The pipeline:
 
-1. fetches the source via the adapter named in `candidate.source`
-2. extracts chapters (EPUB via ebooklib, txt via Gutenberg-header strip +
+1. researches edition provenance and, when safely resolvable, switches from a
+   relay translation to a public-domain original-language source
+2. fetches the resolved source via its adapter
+3. extracts chapters (EPUB via ebooklib, txt via Gutenberg-header strip +
    chapter regex)
-3. translates each chapter in 20-paragraph batches through GLM-5.2
-4. assembles the bilingual EPUB (block-level `<p class="cp-original">` +
+4. extracts book-wide concepts (including lowercase/multiword terms), gathers
+   web evidence, and makes the Korean terminology decisions through separate
+   adjudicator, adversarial-review, and enforcement-schema normalization passes;
+   malformed JSON is repaired automatically and no user approval is requested
+5. translates each chapter in 20-paragraph batches through GLM-5.2, with the
+   concept sheet and semantic-accuracy proofread applied to every batch
+6. independently reviews every final source/translation pair for semantic
+   errors, omissions, untranslated quotations, and malformed Korean; then
+   audits every strict concept occurrence, repairs violations in context, and
+   refuses publication if any issue remains. A normally
+   forbidden Korean string may remain only when an autonomous contextual review
+   records that it translates a different source lemma in that paragraph; a
+   canonical term may be omitted only when an exact longer source expression
+   proves that the matched spelling is a non-conceptual idiom, never for an
+   exact standalone concept occurrence
+7. assembles the bilingual EPUB (block-level `<p class="cp-original">` +
    `<p class="cp-translation">` pairs — the firmware contract)
-5. `calibredb add` into `~/Calibre-Library/`, tagged
+8. `calibredb add` into `~/Calibre-Library/`, tagged
    `bilingual, korean, openclaw`
-6. attempts an HTTP POST push of the EPUB to the device's File Transfer server
+9. attempts an HTTP POST push of the EPUB to the device's File Transfer server
    (`http://crosspoint.local/upload?path=/Books`). If the device isn't in
    File Transfer mode, the push is silently skipped — the book is still
    available via OPDS pull. Push result is recorded in `catalog.json` under
    `device_push`.
-7. sends a LINE completion notice reflecting the delivery path (pushed
+10. sends a LINE completion notice reflecting the delivery path (pushed
    directly, or OPDS pickup instructions)
 
 ### 3. Report progress (if asked)
@@ -114,6 +141,20 @@ cd ~/github/BabelForge
 ./venv/bin/python3 -c "import catalog; print(catalog.recent(5))"
 ```
 
+### 4. Recover a failed request
+
+When the MCP status is `failed`, inspect the recorded error and use
+`retry_request(req_id)` for a retry-safe endpoint/structured-output failure.
+Retry automatically at most once for an unchanged failure, then poll the same
+`req_id`. A new retry is allowed after the underlying code or dependency state
+has materially changed and passed validation. Do not create a second request,
+edit production Python files, or fall back to shell commands merely to resume
+the book.
+
+If a shell command unexpectedly requests approval, do not print `/approve`
+yourself; only the user can grant that approval. Continue through the typed MCP
+tools when possible, otherwise report the exact blocked action.
+
 ## Environment requirements
 
 - `ZAI_API_KEY` exported (GLM Coding Plan). Without it, translate.py fails
@@ -127,7 +168,8 @@ cd ~/github/BabelForge
    Settings → Reader → Font Family.
 2. Settings → System → OPDS Servers → Add Server:
    - Name: `OpenClaw Library`
-   - URL: `https://books.getlingo.store/opds`
+   - URL: `http://books.foundby.kr/opds` (the feed is public; HTTP avoids the
+     verified X3/X4 TLS failure while retaining the stable domain)
    - Username/Password: (none — Content Server runs without auth on the tunnel;
      enable Basic auth later if exposing wider)
 3. Settings → Controls → Long-press Menu → cycle to **Bilingual Toggle**
@@ -141,11 +183,17 @@ to switch views.
 
 - Default auto-suggest never touches Anna's Archive; the `annas:` prefix is
   the only gate. Copyright is the user's responsibility.
+- Precision translation is mandatory. There is no fast-mode flag and no
+  terminology approval pause; source research and multiple evidence-based
+  model passes decide autonomously. Failed research or a remaining strict-term
+  violation blocks publication. A final full-book semantic/orthographic pass
+  must also review every pair and rejects untranslated substantive passages.
 - The firmware parser only honors block-level `<p>` cp-original/cp-translation
   classes; the assembler never emits `<span>` markers.
 - ESP32-C3 can't hold two EPUBs in RAM — single-file bilingual only.
 - 2.4 GHz Wi-Fi only on the device.
-- No completion push to the device by design — the user pulls from OPDS.
+- Device push is best-effort; when the device is unavailable the EPUB remains
+  available through OPDS and the watcher retries queued pushes later.
 
 ## Format contract reference
 

CLAUDE.md

@@ -14,11 +14,16 @@ is **relocatable** (no hardcoded absolute paths — keep it that way).
 
 ```
 request → search (Gutenberg / Standard Ebooks / Anna's / local)
+        → mandatory source research (autonomously prefer a verified public-domain
+                                     original-language edition over relay translation)
         → fetch source → extract chapters (BeautifulSoup / ebooklib)
-        → auto-glossary (recurring names/terms → canonical Korean, web-search-grounded via Z.ai MCP)
-        → GLM translation (paragraph-aligned, char-budget-capped batches, concurrent
-                           GLM workers, 2-pass draft+proofread, HTML-tag preserving,
-                           glossary + preceding-source-context aware)
+        → structured concept sheet (whole-book candidates → web grounding
+                                    → adjudication → adversarial cross-check
+                                    → enforcement-schema normalization)
+        → GLM precision translation (paragraph-aligned, char-budget-capped batches,
+                                     concurrent workers, semantic proofread,
+                                     HTML-tag + concept distinctions preserved)
+        → fail-closed whole-book terminology audit + contextual repair
         → assemble bilingual EPUB (cp-original / cp-translation block-level <p> markers)
         → publish: calibredb add → Calibre Content Server OPDS feed
         → best-effort HTTP push to the XTeink reader (skipped silently if unreachable)
@@ -33,16 +38,18 @@ request → search (Gutenberg / Standard Ebooks / Anna's / local)
 | `config.py` | **All config + paths.** `BASE_DIR` from `__file__`; `DATA_DIR`/`LIBRARY_DIR`/`CACHE_DIR`/`CHECKPOINT_DIR`/`LOGS_DIR`; ZAI/GLM, Calibre, OPDS, device-push settings. |
 | `search.py` | Catalog search → JSONL candidates; `openclaw message send` LINE notify. |
 | `sources/` | Source adapters (`gutenberg`, `standard_ebooks`, `annas_archive`, `local_file`), dispatched via `sources/__init__.py::get_adapter()`. |
-| `extract.py` | Chapter/paragraph extraction. |
+| `extract.py` | Chapter/paragraph extraction. Plain-text Gutenberg sources recover title-case multilingual chapter headings from an explicit indented TOC; headings stay as bilingual units so structural improvements can reuse aligned checkpoints. |
 | `translate.py` | **GLM translation engine** — the biggest file. System prompt is **inline** (literary-translator). Batch output is aligned by **numbered `[k]` markers** (`_parse_numbered_slots` — GLM keeps an explicit numbered list far more reliably than a lone `⟦P⟧` delimiter); missing slots are back-filled by single-paragraph calls, so a partial/collapsed batch keeps its good paragraphs. Plus batching, backoff, checkpoint resume, 2-pass. Also: **inline-HTML tag stashing** (`_stash_tags`/`_restore_tags` swap `<i>/<b>/<em>/…` for PUA placeholders so GLM can't mangle them), **glossary** enforcement in the system prompt, and **source-side context** (each batch gets the preceding 3 source paragraphs — parallel-safe, keeps tone/terminology steady across concurrent workers, `config.TRANSLATE_WORKERS`). |
-| `glossary_builder.py` | Auto-glossary — scans source chapters for recurring proper nouns/terms (`build_glossary_from_chapters`) and resolves their canonical Korean rendering. Default (`enrich_glossary_grounded`) **web-searches the top terms via the Z.ai MCP tool** (`mcp_client.py`) and grounds GLM's extraction in real published usage; `enrich_glossary_with_llm` is the offline fallback (plain guess). `pipeline.py` runs it before translation and passes the result to `translate_book`; also a standalone CLI (`--enrich`, `--no-web-search`) writing `config.GLOSSARY_PATH`. |
+| `precision.py` | **Mandatory precision preflight** — researches original language/title, autonomously replaces a relay catalog source when a verified public-domain original is available, extracts lowercase/multiword and rare lexical-neighbor concepts across the book, gathers web evidence, then runs adjudicator + adversarial critic + enforcement-schema normalization passes. Malformed JSON is repaired autonomously. Writes a structured, request-scoped concept sheet under `data/concepts/`. |
+| `quality_audit.py` | **Fail-closed pre-publication audit** — first independently reviews every final pair for semantic errors, omissions, untranslated quotations, and malformed Korean, then checks every strict source concept occurrence for the canonical Korean term and forbidden variants. It repairs issues through GLM and rejects the book if any remain. A forbidden Korean string may be retained only through an explicit paragraph/concept-scoped contextual review when it translates a different source lemma. A matched spelling may skip the canonical term only when an exact, longer source expression proves a non-conceptual idiom (for example `in Wahrheit` = “실은”); an exact standalone concept term is never exemptible. |
+| `glossary_builder.py` | Legacy standalone flat-glossary utility. The production pipeline no longer uses this best-effort path; `precision.py` owns mandatory structured terminology research. |
 | `mcp_client.py` | Minimal **self-contained MCP (Streamable-HTTP) client** — no Node dependency. Speaks initialize → notifications/initialized → tools/call over urllib against Z.ai MCP servers, authed with the Coding-Plan key. Exposes `web_search(query)`; fails soft (returns `[]`, never raises) so a search hiccup never blocks translation. |
 | `assemble.py` | Builds the bilingual EPUB. Emits **block-level `<p class="cp-original">` / `<p class="cp-translation">` only** — never `<span>` (firmware parser contract). |
 | `publish.py` | `calibredb add` into `~/Calibre-Library`. |
 | `device_push.py` | HTTP POST push to the reader (mDNS/subnet discovery). |
 | `auto_push_watcher.py` + `pending_queue.py` | launchd-driven retry queue for device pushes. |
-| `catalog.py` | `data/catalog.json` state machine (requested → searching → selected → translating → assembling → published/failed). |
-| `babelforge_mcp.py` | **Agent-facing MCP server** (FastMCP) — the appliance's front door. Tools: `search_books` / `translate_book` (detached spawn, returns `req_id` immediately) / `get_status` / `list_recent`. A thin typed skin over the same functions the CLI uses. Runs streamable-HTTP under launchd `com.local.babelforge-mcp` (`http://127.0.0.1:8770/mcp`, `config.MCP_PORT`); `--http` for the appliance, stdio otherwise. Deploy plist: `deploy/com.local.babelforge-mcp.plist`. |
+| `catalog.py` | `data/catalog.json` state machine (requested → researching → selected → analyzing → translating → quality_checked → assembling → published/failed). |
+| `babelforge_mcp.py` | **Agent-facing MCP server** (FastMCP) — the appliance's front door. Tools: `search_books` / `translate_book` (detached spawn, returns `req_id` immediately) / `retry_request` (same-`req_id` checkpoint resume) / `get_status` / `list_recent`. A thin typed skin over the same functions the CLI uses. Runs streamable-HTTP under launchd `com.local.babelforge-mcp` (`http://127.0.0.1:8770/mcp`, `config.MCP_PORT`); `--http` for the appliance, stdio otherwise. Deploy plist: `deploy/com.local.babelforge-mcp.plist`. |
 
 ## Run
 
@@ -71,7 +78,7 @@ BabelForge is self-contained Python, but leans on four things outside it:
 | **`openclaw` CLI** (`/opt/homebrew/bin/openclaw`) | LINE notifications (`search.py`, `config.py:OPENCLAW_BIN`). | One `openclaw message send` subprocess call — no code dependency. |
 | **Calibre Content Server** (launchd `com.local.calibre-server`, port 8080; cloudflared `com.getlingo.openclaw.cloudflared`) | Hosts the OPDS feed at `books.foundby.kr/opds`. `publish.py` only calls `calibredb add`. The older `books.getlingo.store` route may remain as a compatibility alias, but generated BabelForge links use `books.foundby.kr`. | External service (KeepAlive launchd). |
 | **ZAI / GLM API** (`api.z.ai`, model `glm-5.2`) | Translation engine. Key from `.env` (`ZAI_API_KEY` or `GLM_API_KEY`) — **never hardcode**. | External API. |
-| **Z.ai MCP** (`api.z.ai/api/mcp/web_search_prime`, tool `web_search_prime`) | Grounds auto-glossary term renderings in published usage (`mcp_client.py`). Same Coding-Plan key; MCP is entitled on it independently of REST. | External MCP (Streamable HTTP). Optional — glossary degrades to an LLM guess if unreachable. |
+| **Z.ai MCP** (`api.z.ai/api/mcp/web_search_prime`, tool `web_search_prime`) | Grounds source provenance and concept decisions in external evidence (`precision.py`, `mcp_client.py`). Same Coding-Plan key; MCP is entitled on it independently of REST. | External MCP (Streamable HTTP). Individual searches fail soft; source-context adjudication and an adversarial cross-check remain mandatory. |
 
 ## Conventions & gotchas
 
@@ -82,6 +89,10 @@ BabelForge is self-contained Python, but leans on four things outside it:
 - **EPUB format is a contract:** only block-level `<p>` `cp-original`/`cp-translation`; single-file bilingual (ESP32-C3 can't hold two EPUBs in RAM). Don't emit `<span>` markers.
 - **Anna's Archive** is opt-in only via the `annas:` prefix; default search never touches it (copyright = user's responsibility).
 - **Checkpoint resume never regresses:** a re-queued batch window can mix already-translated paragraphs with blank ones (windows are recomputed each run and can shift, e.g. after a batching change), so if that batch ultimately fails, `_commit_batch` in `translate.py` must never overwrite an already non-blank slot with a blank result. This bit us live — a resumed run's rate-limit failures wiped ~90 previously-good translations before the guard was added. Preserve that invariant if you touch `_commit_batch`.
+- **Structural checkpoint remap is exact and audited:** when a better extractor replaces `Untitled` chunks with real chapters, translations may be remapped only by exact in-order source strings at ≥95% coverage. The current semantic and terminology audits always run afterward.
+- **Precision is mandatory:** there is no fast/best-effort production mode. Concept research failures and unresolved strict-term audit failures stop publication. Checkpoints are keyed by both source hash and precision-profile hash, so terminology/prompt changes intentionally restart stale work.
+- **Strict rules are sense-aware but fail closed:** `avoid_ko` is concept-scoped, not paragraph-global, so `Erkenntniss→인식` may coexist with `Wissen→지식`. The auditor records exact paragraph/concept/string exemptions. A canonical term may be absent only for an independently reviewed, verbatim multiword idiom containing the matched spelling; standalone concept occurrences still fail closed.
+- **Structured JSON uses the provider contract:** precision research and the final audit call Z.ai with `response_format=json_object`. Business error `1213` is a request-parameter error (not a length/rate-limit error); never blindly replay the same invalid body. Empty/malformed responses are repaired only with non-empty wrapped input, then regenerated from the original evidence payload.
 - **Gitignored:** `venv/`, `.env`, `data/`, `logs/`, `__pycache__/` (see `.gitignore`).
 
 ## Multi-agent harness