feat(routing): add automatic publication - #1989
Conversation
Signed-off-by: Dennis Lanov <dennis.lanov@gmail.com>
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
tkircsi
left a comment
There was a problem hiding this comment.
@dlanov Thanks for this — the wiring is clean and follows the existing option/config patterns. My concern is the pipeline it feeds into, and I think it's worth waiting for #1967 (DHT-only remote discovery) before going further.
The problem today. Auto-publish creates one publication row and one DHT job per record. The publication pipeline is sized for occasional user-initiated batches: scheduler_interval defaults to 1h, worker_count to 1, and the work queue holds 100 items with a non-blocking send that skips the overflow until the next tick. Importing 1000 records means 1000 PENDING rows dispatched in hourly bursts of ~100, so the last record is announced roughly ten hours after the push. Manual publish never hits this because a query or --all produces one publication covering many CIDs.
Why #1967 changes the picture. It already fixes most of this: the scheduler switches to a blocking send (real backpressure instead of dropping), a wake channel makes a publish take effect immediately rather than at the next interval, autosync is removed (so this no longer re-announces records mirrored from peers), and Publish becomes a durable published flag with a reprovide cycle that enumerates published records, deduplicates labels across them, and provides concurrently.
That last part is the important one: in v2, publishing is setting a flag — the reprovide cycle does the announcing, batched and deduplicated. So auto-publish could reduce to SetRecordPublished(cid, true) at ingest, with no publication rows, no scheduler involvement, and no batching logic to design. That's a fraction of this PR.
One trap if this lands as-is on top of v2: it calls db.CreatePublication directly rather than publicationService.CreatePublication, so it bypasses the new wake() — manual publishes would become near-instant while auto-published records still wait for the hourly tick. (v2 also removes the reason for going to the DB directly: routing.New no longer takes the ingestor, so the publication service can be built before it.)
The two branches also conflict directly — auto_publish is inserted right above the gossipsub: block that #1967 deletes in both values files, and both edit the same routing.New/ingest.New lines in server.go.
Suggest we let #1967 land and then revisit this against the v2 model rather than rebasing mechanically.
Summary
routing.auto_publishserver configurationfalseTesting
golangci-lint run --config .golangci.yml ./...go test ./... -count=1git diff --checkFixes #1966