fix: admit already_stored to the uploads status constraint and delete path - #156
fix: admit already_stored to the uploads status constraint and delete path#156Nic-dorman wants to merge 7 commits into
Conversation
download_cache_private (default off) admits private uploads to the download cache — read-through and seeding, keyed on the serve path's DataMap derivation. Deletes now propagate to every instance: the delete appends the upload's cache keys to cache_purge_log before any row is removed, each instance's 1-minute sweep tick consumes the tail and never advances past a failed unlink, and boot reconciliation validates the whole cache directory against live rows (uploads.cache_key, stamped at terminal statuses and backfilled at writer boot) for instances that were down past log retention. Purge window: synchronous on the handling instance, ~one tick everywhere else. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Boot reconciliation (correctly) purged the sweep tests' fixtures as orphans before the eviction assertions ran — the entries stood in for legitimately cached live content and now have matching rows, which is also what production looks like. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Finding 1: the purge-log append and upload-row delete now commit in one transaction — no interleaving lets a reconciling instance record a log high-water mark, still see the live row, and miss the delete between. A refused delete rolls its log rows back (regression-tested). Finding 2: the full liveness reconciliation now re-runs every 12 hours (and on demand) as the guaranteed backstop, so a purge whose log row was pruned while its unlink was stuck is re-derived from live rows and retried until the bytes are gone; a stuck key no longer delays later purges — every entry is attempted per tick, only the high-water mark waits for the contiguous prefix. Both panel reproductions are now regression tests. Finding 3: reconciliation is policy-aware — turning download_cache_private off purges already-cached private plaintext (next tick online, boot reconciliation for instances that were down) instead of stranding it; no-op setting updates do no spurious work. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Completes the panel's no-op-update matrix — false-to-false was already asserted; both are the same non-transition branch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… path The V2-399 dedup status has been written by the worker since it shipped, but both dialects' schema CHECK constraints never listed it — on any database built from the migrations, a duplicate re-upload failed at the status flip after the network store succeeded. It was also absent from the delete path's allowed statuses, leaving such uploads with no erasure path. Migration 014 extends the CHECK. SQLite needs a table rebuild; renaming uploads rewrites the FK clauses of file_tags, collection_files, and transactions to follow the old table, so those three are rebuilt against the new uploads (the 011 pattern), and the uploads indexes are recreated only after uploads_old is dropped, which is when their names free up. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…status # Conflicts: # internal/services/upload.go
Hermes Agent review — Verdict: No blockers ✅Reviewed at head What this fixes (confirmed)
Checks that held
Non-blocking follow-ups
|
…delete test Review follow-ups from #156: operators with pre-014 databases have dedup re-uploads stuck in failed state that migration 014 does not rewrite — one manual Retry completes them at zero cost. Plus the public-variant delete/purge-fan-out regression. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Both follow-ups handled in the new head:
On the Down-migration row-level assertion: deliberately skipped — this repo's test harness never exercises goose Down migrations, and introducing that machinery for one CASE expression is out of proportion; the fold is documented in the migration and reviewed above. CI re-running on the new head. |
Migration 014 must apply after #155's 013 (goose skips out-of-order versions, and the rebuilt table includes
cache_key), so this branch is based on #155's head. Once #155 merges:gh pr update-branch(or local master-merge) collapses this diff to the final commit.What
The
already_storedstatus (V2-399 content-addressed dedup — a re-upload whose chunks were all already on the network) has been written by the worker since V2-399 shipped, but the schema never allowed it: both dialects' initial CHECK constraints omit it, so on any database built from the migrations the transition fails — the user sees "Failed to save upload record" after the network store succeeded. Unnoticed because e2e always uploads fresh content. Found by the #154 panel review.Second half:
Deleteonly allowedfailed/completed, so analready_storedupload was permanently undeletable — which since #154/#155 also meant no erasure path for its row, DataMap, or cached bytes.How
uploads_status_checkwith the five statuses. SQLite rebuilds the table (CHECKs can't be altered); two sqlite subtleties handled: renaminguploadsrewrites the FK clauses of every referencing table (file_tags,collection_files,transactions— all three rebuilt against the new table, the 011 pattern;legacy_alter_tableis ignored inside goose's transaction), and index names follow the renamed table until it drops, so the uploads indexes are recreated only afteruploads_oldis gone. Down foldsalready_storedintocompleted.Delete's status list gainsalready_stored(error text updated). Quota queries and the web UI already handled the status — verified, no changes needed there.Tests
TestMarkAlreadyStoredPersists— the direct regression: bothMarkAlreadyStoredvariants succeed against the freshly-migrated schema in both CI dialects (fails on the pre-014 constraint), and stampcache_key.TestDeleteAlreadyStoredUpload— deletable, row gone, purge-log fan-out runs like any other delete.-racesuite green in both dialects locally (sqlite) + CI (postgres).🤖 Generated with Claude Code