Conversation
RED: go test -run TestIVF ./pkg/engine/ fails to build — undefined: ivfMinDocs, buildIVF, ivfOrder, ivfNProbe, ivfNList. The compile failure is the intended signal: ivf.go does not exist, so the tests reference the code path that has to.
GREEN: go test -run TestIVF ./pkg/engine/ — 6 tests pass, recall@10 = 1.000 at nprobe=8 of nlist=91 on the clustered corpus. Determinism comes from the absence of a generator rather than a fixed seed: the training sample and the initial centroids are both taken on a fixed stride, so two builds of one corpus agree bit for bit.
RED: go test -run 'TestIVF|TestAV2|TestAV3|TestTheTwo|TestASegmentMixing|TestADamagedIVF|TestAnUnpartitioned' ./pkg/engine/ fails to build — undefined: ivfFile, segment.ivf, segment.nearest. Covers: the section round-trips centroid bits, an unpartitioned segment still writes a constant-size section, a v2 generation opens under the v3 reader and answers exactly, the two readers rank the same, a segment mixing frame versions is refused, a v3 segment missing ivf is refused, and a damaged list costs speed rather than availability.
GREEN: go test ./... all packages pass, including the seven new reproducers and every milestone 2/3a format test unchanged. The version in a segment's meta frame chooses its section list, and every other section has to agree with it. v3 is v2 plus ivf, so a v2 generation opens, reports no partition and offers every id — the same fallback a pending segment and a sub-4096 segment already need, which is what made the two-version reader cheaper than a converter.
RED: go test -run 'TestNearest|TestMergeUpgrades|TestMergeIsByteDeterministicWithVectors|TestMergeRefusesAPartition' ./pkg/engine/ fails to build — Index.Nearest undefined. Covers: at least k candidates whatever the partition's shape, the scan actually narrows, every candidate resolves and none repeats, pending documents are never skipped, a wrong-width query gets everything so the scorer can report ErrDimMismatch, and a merge rewrites a v2 run as v3 deterministically.
…he scorer GREEN: go test ./... all packages pass; make arch green. Nearest returns DocIDs and no score. The engine knows which documents are worth scoring because the partition is a section of the format; how close each one is stays in scorer/vector, so its rules on zero norms, non-finite components and mixed widths are untouched. D-008. Merge doubles as the format converter: it rewrites the run it collapses with the current writer, so a v2 generation becomes v3 through the maintenance an index already performs. One intended golden edit: method Index.Nearest([]float32, int) []DocID.
RED: go test -run TestTheScanIsNarrowedByTheIndex ./pkg/scorer/vector/ fails — 3,669,728 B allocated per query against 2,097,152 B of corpus vectors (175.0%). ponytail:36 said the brute-force scan was milestone 3's to remove; this is the first test in the package that can see it, because an uncommitted index carries no partition and every existing test uses one.
GREEN: go test ./... all pass. The narrowing reproducer goes from 3,669,728 B/query (175.0% of corpus vector bytes) to 338,620 B (16.1%), and all twelve existing contract tests pass unmodified — which is the evidence the repayment changed nothing a caller can observe. The price tag, non-comment lines: 4 removed, 3 added, all in the loop header. Every rule about zero norms, non-finite queries, ErrDimMismatch and context polling stayed exactly where it was, because the metric never left this file. pkg/fusion 0 lines changed
make fuzz green: FuzzSegmentDecoding and FuzzParseSection, 30s each, 6.3M execs, no panics. The section's header sizes two tables and its offset table addresses a payload, so it is three more numbers that come off disk and get acted on before anything vouches for them.
make eval on the pre-existing format v2 index: text+vector 0.6233, identical to the milestone 3a figure, over 171,332 documents. That is the two-version reader working on real data and the repayment leaving the exact path bit-identical. recall measures what nDCG cannot see — a partition that drops neighbours the qrels never judged holds nDCG steady while being wrong. It prints overlap with a brute-force scan, candidates per query, latency both ways, and the per-query working set in bytes and in distinct 4 KiB pages, beside the whole docs section for scale.
The plan fixed the quality bar before anything was built and registered what to do on a miss: raise nprobe and re-measure. nprobe=8 scored text+vector 0.6003 against a 0.6233 baseline. The curve: nprobe 8 16 32 64 128 256 nDCG 0.6003 0.6095 0.6174 0.6211 0.6205 0.6233 At 64: recall@10 0.9920 (was 0.8500), 30,549 candidates per query (17.8%), 4.6x faster than brute force (was 31.2x). Determinism holds: two independent builds of the 148,232-vector partition produce byte-identical ivf sections, sha256 bc042260...b2b6d89. Both narrowing tests move to 65,536-document corpora, because a constant nprobe cannot narrow a segment with fewer lists than nprobe — that is what makes the constant a constant, and a small corpus cannot show it.
FORMAT.md becomes version 3: the ivf section spec, the version-decides-the- section-list rule, four new rejection rows, and section 7.7 — append a section rather than changing one, and the old version stays readable by construction. Three rows milestone 3a had already made false are removed from the limits table. FINDINGS milestone 3b reports both halves. The scan is gone: recall@10 0.992, 4.6x faster, quality inside the bar. The working set is not: 210 MiB per query against a 12 MiB prediction, and the measurement names why — a candidate costs a whole record (1.4x) and a record costs whole pages (1.7x). The plan's registered repayment, a vectors section, would only remove the first. What the second names is centroid-ordered docs, which is a milestone, and the price of it (positional DocIDs, TopK's tiebreak, merge as concatenation) is written down now rather than discovered later. One interval crossed zero and is reported rather than buried: text+vector - text is now +0.0386 [-0.0015, +0.0779]. D-008 records both open decisions and what would falsify each. docs/testing/weft-m3b.tdd.md is the TDD evidence report, including the plan's own internal contradiction about the pass line and how it was resolved.
…reen GOOS=windows go build ./... caught it: syscall.Getrusage does not exist there. Same shape as pkg/engine's mmap split — unix does the real thing, everything else stays buildable and reports 0 rather than substituting a differently-defined number a reader would compare against EVAL's figures. GOOS=windows go build, GOOS=linux GOARCH=386 go build, go vet and the weft-eval tests all pass.
…probe Four findings, all inside the partition. The training stride divided down, so a 39,999-vector corpus trained on documents 0..19,999 and nothing after it — the comment promised an even sample and the arithmetic did not deliver one. Ceiling division now, written 1+(count-1)/ivfSample so a 32-bit build cannot wrap the sum. decodeList sized its output from a count it had just read off disk. A damaged counts table naming 100 million ids makes every list(0) allocate 400 MB and then throw it away on the checksum, and one query decodes up to nprobe lists. Clamped to what the remaining payload can hold, which is the rule decodeTermIndex and decodeDocRecord already follow. scrubSegment handed meta's unverified docCount to scrubIVF, which allocates a seen bitmap of that size. A 20-byte file claiming 2^32-1 documents allocated 4 GB before the docoff cross-check two calls later would have refused it. The check moves above the allocation. ivfMinDocs is derived rather than chosen: 4*nprobe^2, not 4096. The two constants are one decision and drifted apart when nprobe went 8 -> 64. At the old floor nlist = 64 = nprobe, so a query probed every list: count 4,096 8,192 16,384 32,768 65,536 nlist 64 91 128 182 256 share 100% 76% 48% 41% 27% 100% of the segment offered as candidates, for a commit that pays a training pass and an assignment pass. The new floor is the size at which a query first touches half a segment or less. Nothing published moves — the evaluation corpus is one segment of 171,332 documents and partitions under either floor. Test suite pays 2.2 s for the bigger fixtures. And a segment holding no vectors now offers no candidates rather than every id it holds. scorer/vector skips those documents before it compares widths, so they can produce neither a score nor ErrDimMismatch — only a decode of every record in the segment. One text-only batch between two vector ones would otherwise reinstate, on every vector query for as long as the segment lives, the full scan the partition exists to remove.
The extent derivation models the index as one contiguous docs section. That is exact for one segment and wrong for more: DocIDs run on across a directory's segments but the docs files do not, each starting its own payload at offset zero, so a second segment's records would be laid out on top of the first's, counted as sharing pages they cannot share, and the working set would print under the truth. `build` publishes one commit and so one segment; anything else is now refused rather than measured wrong. Three more the same review found. The brute-force pass rebuilt a corpus-sized DocID slice per query, inside the very timing the printed speedup divides by — hoisted, since the index does not change under the loop. A run where no query produced an exact top-k printed NaN for the recall and +Inf for the worst query, which is the mixed-embedding-model case published as a measurement; it is an error now. And MaxRSS is labelled for what it is: a high-water mark both passes reached, not the partition's working set. extents.size was int32 holding an int64 width, which truncates a record past 2 GiB. int64, and the conversion disappears with it. `make recall`'s SKIP branch skipped nothing: each recipe line gets its own shell, so `exit 0` in the first ended that shell and go run followed anyway. One if/else/fi in one shell, the shape `deps` already uses. recall itself is split into vectorQueries, measureRecall and a print method — 73 statements was over the funlen bar and the cyclomatic count was 22 against a limit of 20.
`make lint` has been red since the ivf section landed, and CI pins the same golangci version, so this was not a local disagreement. G115 fires all over ivf.go for the reason the config already records about its siblings: the finding is backwards, firing precisely where the guard is. parseIVF sizes the centroid region in uint64 before it slices anything, decodeList refuses a document id against the segment's own count, and encodeIVF writes lengths buildIVF derived from a corpus Add had already capped. The path regex gains `ivf` rather than a new rule. weft-eval gets its own entry, and a different reason: it measures rather than parses, so every conversion in the extent derivation is a length or a timestamp Go itself produced. uvLen is handed what len() returned and vLen zigzags a Unix time — there is no file on the other side of those to lie about a width. unparam joins the test-file exclusions. A helper parameter every caller passes the same value for is what unparam calls dead flexibility and a test calls its subject: commitGenerations(t, dir, sizes, dim, 12) says at the call site what width the corpus is. maxRSS keeps its two conversions and silences unconvert where the reason can be read: Maxrss is int64 on the 64-bit targets and int32 on linux/386 and linux/arm, so unconvert is right about the machine it ran on and wrong about the build.
FORMAT, DECISIONS and FINDINGS all quoted 4,096 as the size below which no partition is built, and the CHANGELOG entries quoted it twice more. The floor is 16,384 now — 4*nprobe^2 — and FINDINGS carries why, because the move is a second finding about nprobe rather than a tuning change: a segment at the old floor offered 100% of itself as candidates while its commit paid for a partition that excluded nothing. The Nearest entry needed a correction of its own. It promised that documents with no vector are included, which stops being true for a segment holding no vectors at all — every candidate it could offer is one the caller would skip, so offering them buys a decode of the segment and nothing else. FORMAT's nlist formula was missing the clamp buildIVF actually applies: min(ivfNList(count), len(sample)/dim), not ceil(sqrt(count)) alone. A reader reconstructing bytes from the document would have come out short on any segment where most documents carry no vector.
A complexity review raised eight; five held and three did not survive being checked. Nothing here changes observable behaviour, so nothing here had a RED gate — what stands in for one is named per change in the TDD record. - ivfNList: a 16-line integer square root becomes math.Ceil(math.Sqrt(...)). Exhaustively equal to the old body for every count below 5,000,000 and at the powers up to 2^62; IEEE-754 requires a correctly rounded sqrt and maxDocCount is 2^32-1, so a perfect square cannot round up. The test table gained 4095, 2^20, 2^20+1 and maxDocCount. - ivfOrder: four guards deleted and the precondition moved into the doc comment, with segment.nearest's order == nil fallback. parseIVF allocates exactly nlist x dim centroids, and nearest has already returned for a segment with no vectors, no partition, or a query of another width. - workingSet: the page set becomes a running counter, which the ascending id order Nearest guarantees pays for. A new characterization test pins seven layouts and passed against the map before it passed against the counter. - ivf_test: splitmix goes, math/rand/v2 on a fixed seed replaces it — the same stdlib internal/eval already uses. ivf.go still holds no generator at all, which is the claim TestIVFTrainingIsDeterministic actually asserts. - .golangci.yaml: four lines defending the unparam exclusion become one. The exclusion stays; both helpers it covers hold their value as a local constant at the call site, so dropping the parameter would split one fact in two. Withdrawn, and recorded rather than dropped: replacing recordExtents with an average record size cannot produce FINDINGS section 3's numbers (it inverts the page multiplier, 1.94 modelled against 1.68 measured); /usr/bin/time gives one peak where the plan asked for before and after; and ivfListLen cannot go while segWriter streams without a seek. Re-measured, and every published figure reproduces: recall@10 0.9920, 30,549.5 candidates, 124.1 MiB of records, 210.1 MiB of pages, 123 ms against 570 ms. No document needed editing.
github-actions
Bot
added
documentation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 49b4a62af7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…ed it A max-effort review raised fifteen; twelve are here and three are recorded rather than taken. Five are defects a reader or a query can reach. - parseIVF: the centroid fitness check computed nlist*dim*4 in uint64, which wraps, so a meta claiming vecDim = 2^52 beside an ivf header claiming nlist = 1024 passed the check and panicked in make. The bound is a division now — a quotient cannot wrap and nlist >= 1 is already established there. TestIVFHeaderIsRefusedRatherThanTrusted pins five payloads and fails with makeslice against the multiply. - parseIVF: dim == 0 with nlist > 0 was accepted, because the width comparison was against meta's vecDim rather than against zero. No writer can produce it, Scrub called it healthy, and ivfOrder would have tied all 1,024 lists at a dot of zero — a probed set chosen arbitrarily rather than ranked. Refused. - segment.nearest: nprobe counted lists, not lists with members. ivfRefine leaves an unclaimed centroid at the direction it was seeded from, so it ranks high for a query near that document and spent a probe returning nothing; the widening loop is no repair, since it stops at k and k is already met. The budget counts members now, and ivf.go's comment claiming the opposite is gone. - segSectionsFor: segSections[:len-1] encoded "exactly one section was ever added", so a fourth version appending one would have demanded ivf of a 3a segment and refused a directory FORMAT section 1 promises is readable. It is a per-version count, and the returned slice is capped so it no longer shares a backing array with the package-level table — one append through the v2 path would have overwritten the ivf entry process-wide. The rs[6] index becomes an ivfReader lookup by kind. - segment.close: s.ivf was released after the unmap, against the order its own comment and Index.Close document. - ivfNormalizeSum: sq += c*c fuses to one FMADDD on arm64 and stays MULSD+ADDSD on amd64. float64 squared does not fit float64, so the fusion skips a rounding and the norm differs by an ULP between architectures — on the only float64 multiply between a corpus and the bytes FINDINGS publishes a sha256 for. An explicit conversion stops it. ivfDot's fusion is harmless for the reason now written beside it: two widened float32s have 48 significand bits. Four are in the instrument, which matters because its output is published. recordExtents claimed to cross-check its derived total against the docs section on disk and never did, and started record 0 at offset 0 when encodeDocs writes a count uvarint inside a framed section — every page boundary was off. The partitioned pass now runs before the brute-force one, which was warming the page cache for the measurement that follows it; rssBefore is sampled before the walk that decodes the corpus rather than after; and make recall measures the directory it checks. Seven are documents. The per-query cost model in ivf.go, index.go and FINDINGS section 4.3 was arithmetic from nprobe = 8 — 2.7e6 MACs and 42x where the constants give 1.79e7 and 6.4x. FINDINGS section 4.2's "2% at ten million" stops at 6.25%, because ivfMaxList pins nlist at 1,024 twelve lines above it. FORMAT no longer says Merge is the background converter that upgrades v2 (it returns nil below nine segments and weft does not call it), no longer says nlist is reader policy and unstored (it is the section's first uvarint), and states the vector-less segment exception to "every id is a candidate". Four are tests that passed against a mutation. The -1 in the page count and buildIVF's empty-sample guard both had no witness — deleting the guard panics in Commit with a divide by zero, which Add can reach. ivfFile joins the version-mismatch table, and TestNearestNarrowsTheScan commits the 65,536 documents its comment claims instead of 262,144. Re-measured on the evaluation index, and the empty-list fix moves nothing: recall@10 0.9920, worst query 0.8000, 30,549.5 candidates, 124.1 MiB of records, 210.1 MiB of pages, 124 ms against 577 ms. The same run against HEAD gives the same candidate count, so no empty list entered a query's probed set on this corpus — a fact about 148,232 vectors over 414 lists, not a verdict on the guard. FINDINGS section 2 carries that, because a column re-measured after a correctness fix should say so. Not taken. Commit holds the write lock for the whole 68-second training pass, and the repayment is parallelising two per-document argmax loops rather than threading a context — a design change needing its own determinism evidence, so section 4.3 records the ceiling instead. The partition floor counts documents while lists hold only vector-bearing ones, and nprobe is a per-segment budget rather than an index-wide one; both are what FORMAT specifies, and changing either re-measures every published figure.
…measurement An automated review left seven findings on the PR. Three were already answered by the previous commit and are outdated against it — the wrapping centroid bound, the frame header missing from the page offsets, and make recall ignoring EVAL_DATA. Three of the remaining four are here. - weft-eval recall averaged over queries it could not grade. Two shapes reach the loop: a vector of a width the corpus does not carry, which scoreTopK skips every document for and which Nearest answers with the whole segment, and an all-zero vector, which scorer/vector abstains on before it ever calls Nearest. Both leave want untouched, so the guard for a wholly ungradeable run does not fire while one good query remains — and both still contribute a candidate list, a latency and a working set to three figures printed per query. One wrong-width query pulls a corpus-sized candidate count into an average FINDINGS publishes as a percentage of the corpus. They are dropped from every accumulator now and counted in a line the report prints when it has one. - ivfTrainingSample walked its stride and skipped a position with no usable vector without looking for one that has it. At 40,000 documents the stride is 2, so a corpus carrying vectors on odd ids only samples nothing, buildIVF returns no partition, and every vector query over that segment runs the full scan this file exists to remove — indistinguishable from a segment that legitimately holds none. A dense pass now runs when the strided one comes back empty. Only when empty: a short sample is the stride weakness already priced in the comment above it, and falling back on short would change which vectors train the centroids for every corpus with any gap at all. This one cannot change an existing partition, because a corpus that reaches it has none. - parseIVF accepted a centroid of all zeros. It is the NaN rule already in that loop with a quieter face: zero scores zero against every query, so it sorts below any list with a positive dot and its own documents stop being reachable — a recall loss with no error raised, on a section whose checksum is intact. The writer cannot produce one, since a centroid with no members keeps its seeded position and that seed was a normalized sample vector. Not taken: the same finding asked for the centroids to be checked against unit norm, with a tolerance for float32 rounding. Zero is exact and is refused above. A tolerance is a constant with no measurement behind it, and one chosen a shade too tight refuses an index that is fine — the failure the check exists to prevent, pointed the other way. Magnitude alone biases the inner-product ordering; only the absence of magnitude removes a list from it, and that is the case now covered. Each fix fails its test without it, checked by reverting the source and running: the sampler test reports sampling nothing from 20,000 vectors, the centroid test reports parseIVF accepting a zero centroid, and the recall test pins the graded count, the candidate sum and the bytes reached rather than the new field alone. Re-measured on the evaluation index, which still opens: recall@10 0.9920, worst query 0.8000, 30,549.5 candidates, 124.1 MiB of records, 210.1 MiB of pages, all 50 queries graded. gofmt, go vet, golangci-lint at 0 issues, go test ./... and -race on engine and weft-eval.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull Request
Summary
scorer/vectorscored every document in the corpus for every query. On the171,332-document evaluation index that is 171,332 cosine similarities per query and
626.6 MiB of
docstouched — the vector arm was the reason a query cost what it cost.This is milestone 3b. It adds an IVF-flat partition to the segment format, exposes it
as
engine.Index.Nearest, and rewiresscorer/vector's loop to read candidatesinstead of the whole corpus. It also discharges the obligation
FORMAT.md§7.6 placedon a version 3, and it brings an instrument —
weft-eval recall— because nDCG cannottell a good approximation from a bad one.
What changes for a caller: vector results are now approximate. recall@10 against an
exact scan is 0.992,
text+vectornDCG@10 moved 0.6233 → 0.6211, and a query returns4.6× faster. The scorer's contract is unchanged — zero norms, non-finite queries,
ErrDimMismatchand context polling all behave as before, and all twelve of itsexisting tests pass unmodified.
One consequence is not buried: the vector scorer's contribution over text alone was
+0.0407 [+0.0010, +0.0798]and is now+0.0386 [−0.0015, +0.0779]. That intervalcrossed zero. The honest statement after this milestone is that the vector arm's
contribution is not distinguishable from zero on this benchmark. See
docs/EVAL.md§5.14.
Changes
ivf— spherical k-means centroids plus the segment-localDocIDs assigned to each list, every inverted list sealed with a CRC-32C seeded with
its own list number. No RNG: initial centroids are chosen deterministically, so the
same segment always partitions the same way. v2 indexes are read without conversion
(a v2 segment reports no partition); a v2 generation becomes v3 when
Index.Mergerewrites it, which is not an upgrade a caller can count on —
Mergereturnsnilbelow nine segments and weft never calls it for the caller, so a v2 index can stay
v2 indefinitely and answer vector queries by exact full scan. v1 is still refused.
engine.Index.Nearest(v []float32, k int) []DocID— the DocIDs worth scoringexactly, ascending and free of repeats, at least
kwhen the index holds that many.It computes no score: the geometry is the engine's, the metric stays with the caller
(D-008). The result is deliberately wider than the answer — an unpartitioned segment
offers every id it holds, so a caller still skips what it cannot score.
scorer/vectorreadsNearestinstead of scanning — 30,549 candidates of171,332 per query. A segment with no partition is still scored exactly, which is how
the exact path stays reachable without a flag.
weft-eval recallandmake recall— overlap with a brute-force scan, candidatesscored per query, latency both ways, and the working set a query reaches in bytes and
in distinct 4 KiB pages.
nprobe8 → 64, chosen against the pre-registered bar rather than by taste, withthe full sweep published including the part that is inconvenient (the curve is not
monotone — 128 scores below 64).
FORMAT.md§7 for theivfsection,DECISIONS.mdD-008,EVAL.md§5.14,FINDINGS.md, the TDD plan underdocs/testing/, and changie fragments for theformat bump, the API addition, the behaviour change and the new subcommand.
Validation
make vet test—go vet ./...andgo test -race ./..., all packages green.make lint— golangci-lint clean; the exclusions it needs each carry a reasonbeside them in
.golangci.yaml.adding the test that fails without it. New coverage is
pkg/engine/ivf_test.go(765 lines),pkg/engine/nearest_test.go(325),cmd/weft-eval/recall_test.go, andpkg/scorer/vector/narrow_test.go.ivfdecoder entry points are fuzzed (make fuzz) — the section decoder andthe per-list reader, so a corrupt list number cannot reach a slice index.
pkg/engine/testdata/engine_api.txtrecords the one-line API addition; the six readmethods a scorer calls are untouched.
make eval-fullre-ran all five frozen arms andmake recallproduced the recall table. Numbers and how to re-derive them are in
docs/EVAL.md§5.14 and §7.
Review Focus
Nearest's contract (pkg/engine/index.go,ivf.go) — the "wider than theanswer" promise is what lets the unpartitioned, pending, and small-segment cases share
one code path. Worth checking that the fallback really is the same branch and not
three near-copies.
pkg/engine/persist.go,segment.go) — a v2 segment must readcorrectly with no partition, and a v3 segment must not be readable as v2.
two identical lists cannot be swapped undetected. Confirm the seed is what the
decoder verifies against, not just what the writer computed.
candidate floor tracks
nprobe. Both are constants; check they are consistent betweenwriter and reader.
Nearest's doc comment, thechangie fragment and
EVAL.md§5.14 all say the results are approximate. If a callercould still be surprised, that is a bug in the prose.
Risks / Notes
migration is required; v1 is still refused with
ErrBadVersion. An index written bythis branch cannot be read by an earlier build.
scorer/vector: results are approximatewhere they were exact. Nothing in the scorer's error or cancellation contract moved.
nprobeis not configurable. One screw, fixed at a measured value — exposing it wouldinvite tuning against a 50-query benchmark that cannot support the distinction.
says as much about a 50-query benchmark as about the scorer, but it is a real change to
what milestone 4's numbers can claim, and
EVAL.md§5.14 states it rather thanrounding past it.
make recallneeds the built evaluation corpus (make eval-data); it is not part ofthe default test run.
Post-review changes (
598ceac)A review of this branch raised fifteen findings; twelve are applied, three are recorded
as not taken. Everything below is on top of the branch as originally described — the
published numbers did not move, and the one that was at risk was re-measured to prove it.
parseIVF's centroid fitness checkmultiplied in
uint64and wrapped, so a hostile header passed the check and panickedin
make— the bound is a division now, withTestIVFHeaderIsRefusedRatherThanTrustedfailing against the multiply.
dim == 0withnlist > 0was accepted and calledhealthy by
Scrub; refused.segSectionsFor's[:len-1]would misclassify v2 themoment a v4 appends a section, and its result shared a backing array with the
package-level table.
segment.closereleased the partition after the unmap. AndivfNormalizeSum'ssq += c*cfused to oneFMADDDon arm64, making the centroidbytes on disk architecture-dependent on the only
float64multiply in the pathFINDINGSpublishes a sha256 for.nprobecounted lists, not lists with members — an unclaimed centroid stays whereit was seeded, so it ranks high near that document and spent a probe returning nothing.
Fixed, then re-measured: 30,549.5 candidates per query, identical before and after,
with recall@10 0.9920 and the 210.1 MiB working set unchanged. No empty list entered a
query's probed set on this corpus, which is a fact about 148,232 vectors over 414 lists
rather than a verdict on the guard.
FINDINGS§2 says so beside the column.recordExtentsclaimed a cross-check against the
docssection on disk and never performed one, andstarted record 0 at offset 0 where
encodeDocswrites a count uvarint inside a framedsection. The partitioned pass now runs before the brute-force pass that was warming
the page cache for it, and
rssBeforeis sampled before the walk that decodes thecorpus rather than after.
ivf.go,index.goandFINDINGS§4.3 was still arithmetic fromnprobe = 8(2.7e6 MACs and 42× where theshipped constants give 1.79e7 and 6.4×). §4.2's "2% at ten million" floors at 6.25%,
because
ivfMaxListpinsnlistat 1,024.FORMATno longer presentsMergeas abackground converter, no longer calls
nlistunstored reader policy when it is thesection's first uvarint, and states the vector-less segment exception to "every id is a
candidate".
-1in the page count andbuildIVF's empty-sample guard had no witness — deleting the latter panics insideCommitwith a divide by zero, reachable fromAdd.ivfFilejoins theversion-mismatch table, and
TestNearestNarrowsTheScannow commits the 65,536documents its comment claims instead of 262,144.
Not taken, and recorded rather than dropped.
Commitholds the write lock for thewhole 68-second training pass, so every
Search,DocandNearestwaits on it; therepayment is parallelising two per-document argmax loops, which is a design change owing
its own determinism evidence, so §4.3 records the ceiling instead. The partition floor
counts documents while lists hold only vector-bearing ones, and
nprobeis a per-segmentbudget rather than an index-wide one — both are what
FORMATspecifies, and changingeither re-measures every figure above.
Validation:
gofmt,go vet,golangci-lint(0 issues),go test ./...,-race, andFuzzSegmentDecodingto 5.5M execs with no panic.