Optimization: Large library differential scan - #6908
Open
OtterBotSociety wants to merge 5 commits into
Open
Conversation
Detect coarse modtimes and network filesystems per OS, tune adaptive folder checks and directory queues for large libraries, extend SQLite folder queries, and refresh Docker production compose metadata.
Cover CheckFolder adaptive thresholds, directory-queue ordering, unchanged-folder skip paths, walk-time exclusions, folder lookup batching benchmarks, and helpers for coarse-modtime and network volume layouts.
OtterBotSociety
changed the title
OtterBotSociety
commented
May 10, 2026
Comment on lines
+43
to
+95
|
|
||
| // unchangedDirs records directories whose set of direct children has not | ||
| // changed since the last scan. Files directly inside these directories are | ||
| // skipped to avoid unnecessary DB lookups. Subdirectories are still walked | ||
| // recursively so that deeper changes (new files, renamed dirs) are found. | ||
| unchangedDirs sync.Map | ||
|
|
||
| // skipUnchangedFolders is set per-path before walking; false for network | ||
| // filesystems where ModTime may not be reliable. | ||
| skipUnchangedFolders bool | ||
|
|
||
| // dirCheckAttempts and dirCheckHits track the rolling hit rate of | ||
| // CheckFolder calls. shouldCheckFolder uses these to gate calls once the | ||
| // hit rate drops below checkFolderThreshold. | ||
| dirCheckAttempts atomic.Int64 | ||
| dirCheckHits atomic.Int64 | ||
| checkFolderThreshold float64 | ||
| } | ||
|
|
||
| // CheckFolder hit-rate gating — assumptions and context: | ||
| // | ||
| // - After warmup, cumulative hits/attempts proxies whether more CheckFolder | ||
| // calls pay off; walk order can skew this (single global ratio). | ||
| // | ||
| // - A hit is CheckFolder returning unchanged: DB's folder ModTime equals this | ||
| // directory's ModTime from disk (see Scanner.CheckFolder). Skipping enqueue for | ||
| // direct files then assumes that invariant means nothing relevant changed among | ||
| // those children. This is safe for local filesystems, but not for network or | ||
| // coarse-modtime filesystems. | ||
| // | ||
| // - When hits are rare (cold scan / folders absent or changed in DB), extra | ||
| // CheckFolder calls are mostly overhead versus processing dirs/files without this | ||
| // shortcut. | ||
| // | ||
| // - Warmup assumes the first dirCheckWarmup CheckFolder calls are a representative | ||
| // sample of folder outcomes for this walk. It is treated as forecasting the whole | ||
| // job's hit rate. After warmup, a low cumulative rate stops further CheckFolder | ||
| // (e.g. first scan where almost every folder is new). | ||
| // | ||
| // Basic idea: we always CheckFolder until `dirCheckWarmup` folders have been checked, | ||
| // then we continue to do so as long as the hit rate remains >= checkFolderHitRateThreshold | ||
| const ( | ||
| // dirCheckWarmup is the number of CheckFolder calls made before the | ||
| // hit-rate signal is trusted. Keeps the first few directories enabled | ||
| // unconditionally so the estimate is not dominated by noise. | ||
| dirCheckWarmup = 50 | ||
|
|
||
| // checkFolderHitRateThreshold is the minimum hit rate below which | ||
| // CheckFolder is skipped. Derived from benchmarks: break-even is ~0.50 | ||
| // (C_check_windows ≈ 139ms, C_save_per_dir ≈ 280ms); 0.30 is conservative, | ||
| // favouring the warm-scan optimisation in mixed-state libraries. | ||
| checkFolderHitRateThreshold = 0.30 | ||
| ) |
Contributor
Author
There was a problem hiding this comment.
This is the crux of the change -- everything else is the logic that implements the behavior from these annotations
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stash's metadata scan walks configured library folders and reconciles files on disk with the SQLite database (new files, moves, deletions). On large libraries, doing that work repeatedly is expensive.
Current state (
develop@01a75833)An ordinary scan still walks and visits essentially the whole tree, and runs
ScanFilefor each candidate file, but unchanged files (same path modtime/basename, no Rescan) take a cheap path: SQLite lookup plus light bookkeeping, not a full fingerprint re-hash or full decorator/handler pass. Sodevelopalready avoids "re-processing every file" in the "heavy" sense, while elapsed time often still scales with library size because of enumeration and per-file reconciliation.What this PR adds
We extend that modtime pattern to a folder-level shortcut: cheap SQLite
CheckFolderreads (path + stored folder modtime vs disk) with adaptive hit-rate gating let us skip enqueueing files whose parent directory is known unchanged. We also add walk-time pruning of excluded directories, and related fixes (network / coarse-modtime automatic opt-out, hit-rate state reset between jobs). Deep trees are still walked; unchanged parents do not allow skipping descent, because changes deeper in the tree may not bump ancestor mtimes; so this is not "only touch what changed on disk" in the filesystem sense; it reduces queued work and DB-heavy paths when the shortcuts apply.Impact is strongest on normal (non-Rescan) scans of a large, mostly-stable library on local disk, where content changes stay localized so many folders stay unchanged between scans.
Design
CheckFolderwhen the cumulative hit rate suggests those checks are no longer worth it; then processing falls back towarddevelop-like folder handling for subsequent paths in that run.CheckFolderreports an unchanged folder (DB modtime matches disk), we record that folder and omit enqueueing regular files located directly in that folder (subdirectories are still entered so deeper edits remain visible).Parallel metadata scans can race when creating the same folder row.
FolderStore.Createuses insert-or-ignore on the unique folder path and re-reads the existing folder ID when another worker already inserted it, so we avoid SQLite constraint failures without changing how folder rows are modeled.To be clear: the runtime heuristic applies only to whether we even attempt this new optimization. If we're in a regime where most folders have changes, we don't bother continuing to check every folder, as the overhead outweighs the benefit (mostly on first-time library scans)
Expectations: warm rescans on a mostly idle library should spend far less time in parallel file processing (fewer files reach
ScanFile) when many folders are unchanged, while directory listing / walk overhead remains. Adding a handful of new files still walks the tree but should remain closer to "cost driven by touched folders" than "every file pays full queue latency". Rescan and unreliable-modtime filesystems disable the shortcuts so behavior stays conservative.Note on Upgrading
No migration or "onboarding" scan is required. The bookkeeping used here is computed during each scan (it is not a new persisted setting you have to prime). If your library is already indexed in SQLite from normal use, the next metadata scan after upgrading uses the new behavior automatically.
"Rescan"s
If you enable Rescan for a metadata scan, this optimization is turned off for that run: the cheap
CheckFolder/ hit-rate path is not used, and Stash falls back to the same full, conservative directory and file handling Rescan already implied (no skipping files just because a folder looked unchanged). Use a normal scan without Rescan for the fast path.Edge cases
The shortcut hinges on directory modification time:
CheckFolderlooks up the folder by path in SQLite and compares the row's storedModTime(last persisted when Stash wrote that folder record) to the directory's currentmtime. Equal timestamps mean no change to that folder row's notion of the directory, which we use to skip queueing files that are direct children of that directory (not to skip walking into child directories; descendant-only changes may not update the parent's mtime). If directory mtimes are stale, coarse, or inconsistent with what actually changed underneath, relying on them could hide real deltas; so the optimization is only safe when the OS and filesystem type give believable directory timestamps. This is usually the case on typical local filesystems; exceptions are detected and opted out automatically.Note that Stash already assumed accurate modification times before this branch: unchanged-file fast paths on
developcompare each file's path modtime (and related signals) to what SQLite stores, so trusting mtimes for “nothing changed here” is not new. However:Automatic opt-out
We disable the optimization when directory mtimes are likely unreliable: network filesystems and FUSE on Linux (
statfs/ remote volume detection), plus FAT / exFAT (and similar coarse layouts) per OS. For example,statfsmagic on Linux, volume filesystem name on Windows,statfsfstypename on macOS, so removable/USB-FAT-style libraries opt out without manual tuning. If classification fails for a path, we disable the optimization for all subpaths under it (as a conservative but correct default).Benchmarks
Results compare this branch against merge root
develop@01a75833on one machine, one library root (13k files), same sequence every time so numbers are paired (apples-to-apples for each phase).For each build we measure three situations:
Scan settings: Each run uses GraphQL
metadataScanwith Rescan off (default), the benchmark library path as the onlypathsentry, and all scan-time generation disabled (scanGenerateCovers,scanGeneratePreviews,scanGenerateImagePreviews,scanGenerateSprites,scanGeneratePhashes,scanGenerateThumbnails,scanGenerateClipPreviewsallfalse) so timings reflect metadata reconciliation, not thumbnail/preview generation. Both images and videos were included in the scan.The environment was Windows with the library on NTFS (large collection on spinning disk); builds were native Windows binaries. These choices reflect a real desktop + HDD setup, so expect some variance. I'm too lazy to get lab-grade microbenchmarks.
Benchmark results
Windows
.exeon a single (NTFS, HDD) drive, same generated config as above (video_file_naming_algorithm: OSHASH, generation flags off). Warm no-change is the median of five warm samples.develop(ms)develop(human)developmarginally faster)Warm idle rescans were sub-second on the branch vs ~3 s on the parent build; cold was disk-heavy and slightly slower on the branch, but a cold scan only happens once in normal usage.