Skip to content

fix(framework): first-ever converter run should full-sync, not silent… - #9045

Open
bramhanandlingala wants to merge 1 commit into
apache:mainfrom
bramhanandlingala:fix/#9016
Open

fix(framework): first-ever converter run should full-sync, not silent…#9045
bramhanandlingala wants to merge 1 commit into
apache:mainfrom
bramhanandlingala:fix/#9016

Conversation

@bramhanandlingala

@bramhanandlingala bramhanandlingala commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes a bug where a subtask running for the very first time (fresh install, or
a newly-added converter subtask) is silently bootstrapped into incremental
mode, causing it to filter out the entire already-collected backlog and
produce zero rows in the domain layer — even though the tool-layer table is
fully populated.

Repro: fresh DB → GitHub connection + CICD scope → run blueprint once →
_tool_github_runs is populated, but cicd_pipelines returns 0 rows.

Root Cause

bootstrapStateFromCollectorStateIfNeeded (in
backend/helpers/pluginhelper/api/subtask_state_manager.go) copied the
sibling collector's LatestSuccessStart into the subtask's own
PrevStartedAt whenever the subtask had no prior state of its own.

PrevStartedAt == nil is the framework's signal that a subtask has never run
before, and it's what triggers a full sync in
calculateStateManagerIncrementalMode. On a fresh install, the collector
finishes moments before the converter runs — in the same pipeline execution —
so LatestSuccessStart is effectively "now". Bootstrapping PrevStartedAt
from that value made the converter's genuinely-first-ever run look like a
returning run, flipping it into incremental mode and filtering rows by
since against historical timestamps (e.g. github_updated_at), which
silently dropped the entire backlog.

Fix

Stop bootstrapping PrevStartedAt from the collector state. Only TimeAfter
is still inherited from the collector, since it's just a fallback lower bound
and has no effect on the incremental-vs-full-sync decision.

This is a shared framework fix (SubtaskStateManager, used by
StatefulDataConverter/StatefulDataExtractor), so it applies to every
plugin using stateful subtasks — not just GitHub's CICD converters.

Closes #9016

@klesh

klesh commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

The diagnosis looks legitimate to me, but the fix does not; it effectively disables the incremental state completely. I think a better way is to migrate to subtask_state_manager and use the tool layer table (_tool_github_runs) for the Table field so that we can track the incremental state correctly.

@bramhanandlingala

Copy link
Copy Markdown
Contributor Author

Hi @klesh, thanks so much for reviewing this!

Just to clarify the scope of the change — the diff only removes the two lines that copied collectorState.LatestSuccessStart into PrevStartedAt. Everything else stays the same, so incremental mode still works normally after the first run: Close() saves PrevStartedAt, and the next run picks it up and goes incremental as before. It's only the very first run that behaves differently now — it does a proper full sync instead of being mistaken for a run that already happened.

On the _tool_github_runs idea — I think we might run into the same issue there. The extractor writes to that table just before the converter runs, in the same pipeline, so on a genuine first run it would already have data too, and we could end up dropping the backlog again the same way.

Let me know your thoughts, and happy to adjust if I'm missing something on your end!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug][framework] First ever converter run bootstraps into incremental mode and silently skips the entire collected backlog

3 participants