Milestone 3b: the vector arm stops scanning the corpus, and says what that costs

.golangci.yaml

@@ -96,15 +96,17 @@ linters:
       - builtin$
     rules:
       # Test files are allowed the shapes that make a test readable: an ignored
-      # error on a cleanup call, a constant repeated across table rows, and a
-      # long table row.
+      # error on a cleanup call, a constant repeated across table rows, a long
+      # table row, and a helper parameter every caller passes the same value for —
+      # `dim` names the corpus width at the call site of the test that turns on it.
       - linters:
           - errcheck
           - gosec
           - goconst
           - funlen
           - gocyclo
           - lll
+          - unparam
         path: _test\.go$
 
       # G115 flags every width change in the decoder, and in the encoder that
@@ -121,10 +123,27 @@ linters:
       # Writing, every id and count converted is a position in a collection Add
       # has already capped at maxDocCount — a DocID that did not fit is a
       # document the index refused to hold.
+      #
+      # ivf.go is the same file in both directions: parseIVF bounds the centroid
+      # width by dividing what is left of the payload rather than multiplying by it
+      # — a product of nlist and a width meta ranged only against maxInt wraps, and
+      # G115 does not see a multiplication — decodeList refuses a document id
+      # against the segment's own count, and encodeIVF writes nlist, dim and list
+      # lengths that buildIVF derived from a corpus already capped.
+      - linters:
+          - gosec
+        text: "G115"
+        path: pkg/engine/(segment|segments|index|merge|persist|seek|ivf)\.go$
+
+      # weft-eval measures rather than parses, and every conversion in its extent
+      # derivation is a length or a timestamp that Go itself produced: uvLen is
+      # handed what len() returned, and vLen hands a Unix time to PutVarint. There
+      # is no file on the other side of these to lie about a width — a negative
+      # length here would be a broken runtime, not a doctored index.
       - linters:
           - gosec
         text: "G115"
-        path: pkg/engine/(segment|segments|index|merge|persist|seek)\.go$
+        path: cmd/weft-eval/recall\.go$
 
       # decodeDocs and decodePostings are branchy and long because every branch
       # is a corruption check on a file weft did not write. Splitting them to

CHANGELOG.md

@@ -21,15 +21,28 @@ Entries are written with [changie](https://changie.dev) as the change is made, n
 ### Exported API
 
 - Baseline. The surface at this tag is whatever pkg/engine/testdata/engine_api.txt and public_api.txt record at the tagged commit; every later release states its diff against them rather than restating the whole surface. ([#9](https://github.com/skyoo2003/weft/issues/9))
+- Three additions, no changes: Scrub(dir) verifies every byte of a committed index, which Open no longer does; Index.Close releases the mappings an opened index holds; Index.Merge collapses the oldest segments once the count passes eight. The six read methods a scorer calls are untouched, which is the milestone 3 claim. ([#9](https://github.com/skyoo2003/weft/issues/9))
+- engine.Index.Nearest(v []float32, k int) []DocID returns the DocIDs worth scoring exactly for a query vector, at least k of them when the index holds that many vectors. It computes no score: the engine knows which documents are geometrically plausible, and the metric stays with the caller (docs/DECISIONS.md D-008). The result is ascending, free of repeats, and wider than the answer — a segment with no partition offers every id it holds, documents carrying no vector included, so a caller still skips what it cannot score. The one thing left out is a segment holding no vectors at all: every candidate it could offer is one the caller would skip, so offering them would only buy a decode of the segment. It is not a promise that the exact top k are among them; docs/EVAL.md section 5.14 publishes the measured recall against a brute-force scan. ([#11](https://github.com/skyoo2003/weft/issues/11))
 
 ### On-disk format
 
 - Version 1, described in docs/FORMAT.md. Nothing to migrate from — this is the first tag. ([#9](https://github.com/skyoo2003/weft/issues/9))
+- Version 2. Version 1 indexes are refused with ErrBadVersion and are not migrated — weft has no users and the only v1 directory in existence is rebuildable, which is an argument available exactly once (docs/DECISIONS.md D-007). New sections docoff and keys let a DocID or a Key reach its document without decoding the documents in front of it; per-unit checksums let one record, block or entry be verified without reading the whole file; the manifest's segment entries now carry (name, base, count). ([#9](https://github.com/skyoo2003/weft/issues/9))
+- Version 3. One new section, ivf, holding an IVF-flat partition of a segment's vectors: spherical k-means centroids and the segment-local DocIDs assigned to each, each inverted list sealed with a CRC-32C seeded with its own list number. Version 2 indexes are read without conversion — a v2 segment reports no partition and answers with every id it holds, which is the same fallback a pending segment and a segment below 16,384 documents already need, so it costs a branch that had to exist anyway. Index.Merge upgrades a v2 generation to v3 as a side effect of the maintenance an index already performs. Version 1 is still refused. This discharges the obligation FORMAT.md section 7.6 placed on a version 3 (docs/DECISIONS.md D-008). ([#11](https://github.com/skyoo2003/weft/issues/11))
 
 ### Minimum Go
 
 - 1.26. This is the floor, not a tested ceiling — newer is untested rather than unsupported, and the gate pins this one line rather than a matrix. ([#9](https://github.com/skyoo2003/weft/issues/9))
 
+### Added
+
+- `weft-eval recall` and `make recall` measure what nDCG cannot see about an approximate vector index: overlap with a brute-force scan, candidates scored per query, latency both ways, and the working set a query actually reaches in bytes and in distinct pages. A partition that loses half the true neighbours holds nDCG steady when the neighbours it lost were unjudged, which on 50 queries against 171,332 documents is most of them. ([#11](https://github.com/skyoo2003/weft/issues/11))
+
+### Changed
+
+- Open maps segments instead of decoding them, so opening an index costs the vocabulary rather than the corpus — 54 ms against 979 ms on a 171,332-document index. Commit writes only what was added since the last one; on a 7.2 MB corpus a commit after one Add writes 245 bytes, and previous generations are left untouched. Whole-file verification moved from Open into Scrub: damage inside a unit is still refused when that unit is read, and damage in bytes no decoder reaches now needs Scrub to find. ([#9](https://github.com/skyoo2003/weft/issues/9))
+- scorer/vector no longer scans the whole corpus. Its loop now reads engine.Index.Nearest, which on the evaluation corpus scores 30,549 documents of 171,332 per query and returns a query 4.6 times faster. The consequence a caller has to know about is that vector results are now approximate: recall@10 against an exact scan is 0.992 and nDCG@10 for the text+vector arm moved 0.6233 to 0.6211. Nothing about the scorer's contract changed — zero norms, non-finite queries, ErrDimMismatch and context polling all behave exactly as before, and all twelve of its existing tests pass unmodified. A segment with no partition is still scored exactly. ([#11](https://github.com/skyoo2003/weft/issues/11))
+
 ### Security
 
 - Commit now creates its index directory and segment directories with mode 0o700 rather than 0o755. The corpus is the caller's data and nothing weft does needs another user on the machine to read it — including the caller's own group, which 0o750 would let read the 0o644 segment files inside. ([#9](https://github.com/skyoo2003/weft/issues/9))

Makefile

@@ -1,6 +1,6 @@
 .PHONY: all fmt build vet test lint lint-docs spdx fuzz arch deps run example clean \
 	changelog changelog-new changelog-check docs-site release-check \
-	eval eval-full eval-data
+	eval eval-full eval-data recall
 
 # `all` needs nothing installed beyond the Go toolchain, which is what lets a
 # first-time contributor run the whole gate before they have read anything.
@@ -160,6 +160,23 @@ EVAL_DATA ?= .eval-data
 eval:
 	go run ./cmd/weft-eval run
 
+# Milestone 3b. What the approximate vector index costs and what it buys, which
+# `eval` cannot say: nDCG is blind to a partition dropping neighbours the qrels
+# never judged, so recall against a brute-force scan is measured separately.
+#
+# Skipped rather than failed without the data, unlike `eval`. This target exists
+# to be run by anyone reproducing the FINDINGS numbers, and a missing multi-
+# gigabyte download is not a broken checkout.
+#
+# One shell, not two: each recipe line gets its own, so an `exit 0` in the first
+# skips nothing that follows it. Same shape as `deps` above.
+recall:
+	@if [ ! -d $(EVAL_DATA)/index ]; then \
+		echo "SKIP: no index at $(EVAL_DATA)/index — run 'make eval-data' first"; \
+	else \
+		go run ./cmd/weft-eval recall -data $(EVAL_DATA); \
+	fi
+
 # Everything milestone 4 publishes: the degeneracy diagnostic, the frozen arms, the
 # sensitivity sweep, and the fusion weight sweep behind the README's claim that no
 # weight makes the graph stream worth anything. Slower — the sweep alone re-measures

changes/unreleased/Added-20260817-000003.yaml

@@ -0,0 +1,9 @@
+kind: Added
+body: '`weft-eval recall` and `make recall` measure what nDCG cannot see about an approximate
+  vector index: overlap with a brute-force scan, candidates scored per query, latency both
+  ways, and the working set a query actually reaches in bytes and in distinct pages. A
+  partition that loses half the true neighbours holds nDCG steady when the neighbours it
+  lost were unjudged, which on 50 queries against 171,332 documents is most of them.'
+time: 2026-08-17T00:00:03.000000000+09:00
+custom:
+  Issue: 11

changes/unreleased/Changed-20260817-000002.yaml

@@ -0,0 +1,12 @@
+kind: Changed
+body: 'scorer/vector no longer scans the whole corpus. Its loop now reads engine.Index.Nearest,
+  which on the evaluation corpus scores 30,549 documents of 171,332 per query and returns
+  a query 4.6 times faster. The consequence a caller has to know about is that vector
+  results are now approximate: recall@10 against an exact scan is 0.992 and nDCG@10 for
+  the text+vector arm moved 0.6233 to 0.6211. Nothing about the scorer''s contract changed
+  — zero norms, non-finite queries, ErrDimMismatch and context polling all behave exactly
+  as before, and all twelve of its existing tests pass unmodified. A segment with no
+  partition is still scored exactly.'
+time: 2026-08-17T00:00:02.000000000+09:00
+custom:
+  Issue: 11

changes/unreleased/Exported-API-20260817-000001.yaml

@@ -0,0 +1,14 @@
+kind: Exported API
+body: 'engine.Index.Nearest(v []float32, k int) []DocID returns the DocIDs worth scoring
+  exactly for a query vector, at least k of them when the index holds that many vectors.
+  It computes no score: the engine knows which documents are geometrically plausible,
+  and the metric stays with the caller (docs/DECISIONS.md D-008). The result is ascending,
+  free of repeats, and wider than the answer — a segment with no partition offers every
+  id it holds, documents carrying no vector included, so a caller still skips what it
+  cannot score. The one thing left out is a segment holding no vectors at all: every
+  candidate it could offer is one the caller would skip, so offering them would only
+  buy a decode of the segment. It is not a promise that the exact top k are among them;
+  docs/EVAL.md section 5.14 publishes the measured recall against a brute-force scan.'
+time: 2026-08-17T00:00:01.000000000+09:00
+custom:
+  Issue: 11

changes/unreleased/On-disk-format-20260817-000000.yaml

@@ -0,0 +1,13 @@
+kind: On-disk format
+body: 'Version 3. One new section, ivf, holding an IVF-flat partition of a segment''s
+  vectors: spherical k-means centroids and the segment-local DocIDs assigned to each,
+  each inverted list sealed with a CRC-32C seeded with its own list number. Version
+  2 indexes are read without conversion — a v2 segment reports no partition and answers
+  with every id it holds, which is the same fallback a pending segment and a segment
+  below 16,384 documents already need, so it costs a branch that had to exist anyway.
+  Index.Merge upgrades a v2 generation to v3 as a side effect of the maintenance an
+  index already performs. Version 1 is still refused. This discharges the obligation
+  FORMAT.md section 7.6 placed on a version 3 (docs/DECISIONS.md D-008).'
+time: 2026-08-17T00:00:00.000000000+09:00
+custom:
+  Issue: 11

cmd/weft-eval/main.go

@@ -85,6 +85,7 @@ const (
 	cmdRun      = "run"
 	cmdSweep    = "sweep"
 	cmdWeights  = "weights"
+	cmdRecall   = "recall"
 )
 
 func main() {
@@ -113,6 +114,8 @@ func main() {
 		err = sweep(ctx, args)
 	case cmdWeights:
 		err = weights(ctx, args)
+	case cmdRecall:
+		err = recall(ctx, args)
 	case "-h", "--help", "help":
 		usage()
 	default:
@@ -129,7 +132,7 @@ func main() {
 }
 
 func usage() {
-	fmt.Fprint(os.Stderr, `usage: weft-eval <prepare|build|diagnose|run|sweep|weights> [flags]
+	fmt.Fprint(os.Stderr, `usage: weft-eval <prepare|build|diagnose|run|sweep|weights|recall> [flags]
 
   prepare   join corpus ids to Semantic Scholar for citation edges and SPECTER
             vectors. Slow (hours, rate limited) and resumable — rerun to continue.
@@ -140,6 +143,10 @@ func usage() {
   weights   discount the graph stream in fusion and see whether that recovers it.
             Weights attach to stream position, not scorer kind, so fusion still
             does not know what it is holding.
+  recall    what the approximate vector index costs and buys: overlap with a
+            brute-force scan, candidates per query, and the working set a query
+            actually reaches. nDCG cannot see a partition losing neighbours the
+            qrels never judged; this can.
 
 Run any subcommand with -h for its flags. docs/EVAL.md is the measurement design.
 `)