perf: bound indexing memory and flush partial progress on interrupt - #159
Open
iggy wants to merge 1 commit into
Open
perf: bound indexing memory and flush partial progress on interrupt#159iggy wants to merge 1 commit into
iggy wants to merge 1 commit into
Conversation
iggy
force-pushed
the
perf/bound-indexing-memory
branch
from
August 16, 2026 22:25
4f0c4f0 to
e3d6566
Compare
Indexing was O(n^2) in wall time and peaked at file-size memory: each Upsert ran a full-table-scan SELECT (no index, ql ignores it for FirstRow) and hashFile called os.ReadFile, loading the whole file. Changes: - hashFile now streams: imohash via SumSectionReader (fixed samples) and xxh3 via io.Copy into xxh3.New(). Peak memory is bounded by the read buffer, not file size. Rewind the SectionReader between passes (SumSectionReader leaves it at the tail). Hashes are byte-identical to before (TestHashFileMatchesReadFileBasis), so DB rows stay valid. - index.RunCtx loads the host's rows once (LoadExisting) into a read-only snapshot shared by the walk (quick-skip) and consumer (insert-vs-update) without locking, eliminating per-row SELECTs. - Consumer batches writes into single transactions (BatchRows default 1000 / BatchBytes default 4 GiB): ~1000x faster than per-row commits, and a Ctrl-C forfeits at most one in-flight batch. - Walk bounds concurrency by both file count (Workers) and bytes in flight (MaxBytes, default 256 MiB/worker) so a few huge files cannot starve small-file parallelism. - main.run cancels the run's context on SIGINT/SIGTERM so the consumer flushes its in-flight batch (committing partial progress) before closing the store; a second SIGINT exits 130 immediately. - LoadExisting scopes by hostname so a multi-host DB cannot leak cross-host rows into the path-keyed snapshot.
iggy
force-pushed
the
perf/bound-indexing-memory
branch
from
August 16, 2026 22:37
e3d6566 to
81bcb88
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
gocate -updatedbwas slow and memory-hungry on large trees. Profiling showed the cost was not hashing — it was the store write path:UpsertranSELECT ... FirstRow()with no usable index → a full table scan per file (O(n²) total).modernc.org/qlignores the index forFirstRow(verified: 136 ms/row at 200k rows, 1.0× with index).hashFilecalledos.ReadFile, so peak memory = file size.BEGIN TRANSACTION; … COMMIT;.Result: on a 32-core box it sat at ~120% CPU because the hash workers finished fast but funneled into one consumer doing slow sequential scans.
Changes
internal/index/hash.go— stream instead of slurp:imohashviaSumSectionReader(fixed samples) andxxh3viaio.Copyintoxxh3.New(). Peak memory is now bounded by the read buffer. TheSectionReaderis rewound between passes (it's left at the tail after sampling). Hashes are byte-identical to before, so existing DB rows stay valid.internal/index/index.go—RunCtxloads the host's rows once into a read-only snapshot shared by the walk (quick-skip) and consumer (insert-vs-update) with no locking, eliminating the per-row SELECTs. The consumer batches writes into single transactions (default 1000 rows / 4 GiB). The walk bounds concurrency by both file count (Workers) and bytes in flight (MaxBytes, default 256 MiB/worker) so a few huge files can't starve small-file parallelism.cmd/gocate/main.go— cancels the run's context on SIGINT/SIGTERM so the consumer flushes its in-flight batch (committing partial progress) before close; a second SIGINT exits 130.internal/store/store.go—LoadExisting(hostname-scoped) +WriteBatch(one transaction for many rows, rollback on error).Upsertreuses the batch path.Tests
TestHashFileMatchesReadFileBasis— proves streaming hashes equal the oldos.ReadFileapproach across small/medium/large files.TestLoadExistingScopesByHost— proves a multi-host DB can't leak cross-host rows into the path-keyed snapshot.go vetclean.Verification (5.2 GB, 4530 files)
And a SIGINT mid-run commits partial progress (verified: 20k rows after interrupt, DB reopens cleanly).