Skip to content

feat(routing): add automatic publication - #1989

Open
dlanov wants to merge 2 commits into
agntcy:mainfrom
dlanov:feat-1966-routing-auto-publish
Open

feat(routing): add automatic publication#1989
dlanov wants to merge 2 commits into
agntcy:mainfrom
dlanov:feat-1966-routing-auto-publish

Conversation

@dlanov

@dlanov dlanov commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add the opt-in routing.auto_publish server configuration
  • automatically enqueue successfully ingested records for asynchronous publication
  • keep publication enqueue failures non-fatal so ingest remains successful
  • attempt auto-publication after the content-store push and independently of search indexing
  • expose the setting in Helm values with a default of false
  • add focused config, ingest, and publication-request tests

Testing

  • golangci-lint run --config .golangci.yml ./...
  • go test ./... -count=1
  • focused auto-publish, ingest, and configuration tests
  • git diff --check

Fixes #1966

Signed-off-by: Dennis Lanov <dennis.lanov@gmail.com>
@dlanov
dlanov requested a review from a team as a code owner August 7, 2026 00:03
@github-actions github-actions Bot added the size/M Denotes a PR that changes 200-999 lines label Aug 7, 2026
@codecov

codecov Bot commented Aug 7, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 85.71429% with 5 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
server/server.go 72.2% 4 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

@tkircsi tkircsi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dlanov Thanks for this — the wiring is clean and follows the existing option/config patterns. My concern is the pipeline it feeds into, and I think it's worth waiting for #1967 (DHT-only remote discovery) before going further.

The problem today. Auto-publish creates one publication row and one DHT job per record. The publication pipeline is sized for occasional user-initiated batches: scheduler_interval defaults to 1h, worker_count to 1, and the work queue holds 100 items with a non-blocking send that skips the overflow until the next tick. Importing 1000 records means 1000 PENDING rows dispatched in hourly bursts of ~100, so the last record is announced roughly ten hours after the push. Manual publish never hits this because a query or --all produces one publication covering many CIDs.

Why #1967 changes the picture. It already fixes most of this: the scheduler switches to a blocking send (real backpressure instead of dropping), a wake channel makes a publish take effect immediately rather than at the next interval, autosync is removed (so this no longer re-announces records mirrored from peers), and Publish becomes a durable published flag with a reprovide cycle that enumerates published records, deduplicates labels across them, and provides concurrently.

That last part is the important one: in v2, publishing is setting a flag — the reprovide cycle does the announcing, batched and deduplicated. So auto-publish could reduce to SetRecordPublished(cid, true) at ingest, with no publication rows, no scheduler involvement, and no batching logic to design. That's a fraction of this PR.

One trap if this lands as-is on top of v2: it calls db.CreatePublication directly rather than publicationService.CreatePublication, so it bypasses the new wake() — manual publishes would become near-instant while auto-published records still wait for the hourly tick. (v2 also removes the reason for going to the DB directly: routing.New no longer takes the ingestor, so the publication service can be built before it.)

The two branches also conflict directly — auto_publish is inserted right above the gossipsub: block that #1967 deletes in both values files, and both edit the same routing.New/ingest.New lines in server.go.

Suggest we let #1967 land and then revisit this against the v2 model rather than rebasing mechanically.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/M Denotes a PR that changes 200-999 lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature]: Add routing.auto_publish config

3 participants