Skip to content

fix: repair schema v19 drift - #73

Merged
teedole merged 4 commits into
mainfrom
teedole/schema-v19-repair
Jul 30, 2026
Merged

fix: repair schema v19 drift#73
teedole merged 4 commits into
mainfrom
teedole/schema-v19-repair

Conversation

@teedole

@teedole teedole commented Jul 29, 2026

Copy link
Copy Markdown
Member

Summary

  • Bumps the archive baseline to v19 and safely replays every late v18 review repair for archives that had already recorded the earlier v18: recommendation impact columns, idx_block_tool_use, and timestamp-fallback tool duration backfill.
  • Adds semantic owned-schema fingerprinting, actionable bounded drift errors, contiguous migration-history checks, concurrency-safe archive initialization/WAL negotiation, and an HTTP schema_drift conflict response.
  • Freezes a complete v8 schema fixture and proves fresh and migrated archives converge semantically without rebuilding user data.

Why

A dogfooding archive recorded migration v18 before the final review edits landed, so its migration history said v18 while required columns and an index were absent. Watcher syncs and analytics reports then failed repeatedly with raw SQLite errors.

The repair must also distinguish real drift from harmless SQLite representation differences caused by ALTER TABLE, FTS renames, and the historical impact_label_checked default. The new manifest compares schema semantics while preserving SQL literal contents and reporting exact missing or changed objects.

User and developer impact

  • Affected v18 archives self-repair on the next open without deleting source logs or operator-authored recommendation state.
  • Non-repairable drift fails once with a bounded, actionable error that tells the operator to back up before rebuilding.
  • Concurrent first opens no longer race schema creation or WAL negotiation.
  • Future committed migrations are explicitly frozen; review-time schema changes require a new version.

Validation

  • just check passes: 651 tests, TypeScript, Biome (154 files), and distribution staging smoke.
  • New behavior has focused tests. No existing test was weakened or deleted to make this pass.
  • Adversarial review reproduced the original pre-review-v18 lineage and fresh-open races; all valid findings were fixed and re-reviewed with no remaining findings.

Focused coverage includes full v8-to-v19 equivalence, exact pre-review-v18 repair and data preservation, rollback on unrelated drift, malformed/gapped histories, bounded hostile identifiers, SQL-literal-sensitive fingerprints, schema error routing, and 10 barrier-synchronized two-process initialization races.

Archive schema

The baseline is LATEST_SCHEMA_VERSION in src/db.ts. Check one:

  • This change does not touch the archive schema.
  • This change bumps the schema. Existing v8-v18 archives migrate to v19; affected v18 archives receive guarded additive repairs and safe NULL-only metadata backfill before the v19 stamp commits.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the database schema baseline to v19 and introduces robust schema drift detection by generating and comparing canonicalized schema manifests. It also refactors database initialization and migration to handle concurrent openers safely. Feedback on these changes highlights a potential concurrency bottleneck where Atomics.wait is used on the main thread during WAL mode negotiation, suggesting a read-only check of the journal mode first to avoid write contention. Additionally, performance improvements are recommended in the schema manifest generation to replace inefficient JSON.stringify serialization inside sorting comparators with a direct, type-safe comparison function for foreign keys.

Comment thread src/db.ts
Comment thread src/schema-manifest.ts Outdated
Comment thread src/schema-manifest.ts Outdated

@teedole teedole left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed review findings; inline replies include the implementation and validation evidence.

Comment thread src/schema-manifest.ts Outdated
Comment thread src/db.ts

@teedole teedole left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed review findings; inline replies include the implementation and validation evidence.

Comment thread src/schema-manifest.ts Outdated
@teedole
teedole marked this pull request as ready for review July 29, 2026 23:33
@teedole

teedole commented Jul 30, 2026

Copy link
Copy Markdown
Member Author

Validator follow-up complete at signed tip 52325c8.

  • Additive operator-owned schema objects now emit one bounded warning; missing or changed owned objects remain fatal.
  • Migration cleanup preserves the original failure if ROLLBACK also fails.
  • The v19 duration repair skips NULL-to-NULL updates, avoiding WAL churn.
  • Migration tests now remain valid when a later stacked PR raises LATEST_SCHEMA_VERSION.

Evidence: focused database suite 38/38; combined #73-#79 gate 781/781 plus TypeScript, Biome, and distribution staging smoke.

@teedole
teedole merged commit 4e565ef into main Jul 30, 2026
6 checks passed
@teedole
teedole deleted the teedole/schema-v19-repair branch July 30, 2026 02:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant