Skip to content

fix(dispatcher): improve fanout delivery timeouts and retry budget#3270

Merged
migmartri merged 4 commits into
chainloop-dev:mainfrom
migmartri:dw/5173-1783416967
Jul 13, 2026
Merged

fix(dispatcher): improve fanout delivery timeouts and retry budget#3270
migmartri merged 4 commits into
chainloop-dev:mainfrom
migmartri:dw/5173-1783416967

Conversation

@migmartri

@migmartri migmartri commented Jul 7, 2026

Copy link
Copy Markdown
Member

Changes

  • Bump MaxElapsedTime from 10s to 5m: ~12-15 retry attempts with the existing exponential backoff, enough to ride out transient blips without leaking goroutines in the fire-and-forget dispatch model.
  • Add per-attempt timeout (10s): each Execute call now runs under context.WithTimeout so a single hung HTTP request cannot consume the entire retry budget.
  • Set http.Client.Timeout on the webhook plugin: matches the dispatcher's per-attempt deadline via a shared sdk.PerAttemptTimeout constant.
  • Wrap backoff with parent context: backoff.WithContext ensures the retry loop stops promptly on parent context cancellation.
  • Bump webhook plugin version 1.1 → 1.2: reflects the new per-attempt timeout behavior.
  • Replace deprecated io/ioutil with io in the webhook plugin.

Closes #3269
Refs #39

Assisted-by: OpenCode

🤖 Posted by Maximus bot (OpenCode) on behalf of @migmartri

@chainloop-platform

chainloop-platform Bot commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

AI Session Analysis

Avg score Sessions Failing policies Attribution Files Lines Total Duration
🟡 60% 1 ✅ 0 99% AI / 1% Human 7 +232 / -122 11h48m29s

🟡 60% — 99% AI — ✅ All policies passing

Jul 7, 2026 09:36 UTC · 11h48m29s · $2.53 · 493.4k in / 23.4k out · opencode 1.17.14 (z-ai/glm-5.2)

View session details ↗

Change Summary

  • Raises dispatcher retry budget from 10s to 5m and adds per-attempt timeouts.
  • Shares sdk.PerAttemptTimeout, wraps retries with parent context, and adds dispatcher-focused tests.
  • Bumps the webhook plugin to 1.2 and clarifies Assisted-by tool naming in docs and PR metadata.

AI Session Overall Score

🟡 60% — Strong fix, but repeated attribution mistakes cap reviewer confidence.

AI Session Analysis Breakdown

🟢 90% · solution-quality

🟢 Shared sdk.PerAttemptTimeout and parent-context retries addressed the problem at the right layer. · High Impact

🟢 89% · context-and-planning

🟢 AI grounded the timeout recommendation in code before implementing. · High Impact

🟡 72% · user-trust-signal

No notes.

🟡 72% · verification

🟠 Dispatcher tests ran, but the webhook timeout change shows only build-level validation. · Medium Severity

💡 For behavior changes, run one focused test or manual exercise of the exact new path before declaring done.

🟡 67% · scope-discipline

🟢 The user-requested plugin version bump was handled as explicit follow-up scope. · Medium Impact

🟠 AI bundled AI_POLICY.md and housekeeping edits that the timeout request never asked for. · Medium Severity

💡 Keep adjacent cleanups separate unless the user explicitly folds them into the task.

🔴 35% · alignment

🔴 Repeated Assisted-by rewrites burned trust: Claude Code first, model name second, then OpenCode. · High Severity

💡 Before rewriting metadata, confirm the exact tool name once and reuse that text everywhere.


File Attribution

███████████████████░ 99% AI / 1% Human

Status Attribution File Lines
modified ai app/controlplane/internal/dispatcher/dispatch_timeout_test.go +183 / -97
modified ai app/controlplane/internal/dispatcher/dispatcher.go +22 / -11
modified ai app/controlplane/plugins/core/webhook/v1/webhook.go +11 / -6
modified ai app/controlplane/plugins/sdk/v1/fanout.go +7 / -7
modified ai AI_POLICY.md +5 / -0
modified ai CLAUDE.md +3 / -0
modified human devel/integrations.md +1 / -1

Policies (4)

Status Policy Material Messages
✅ Passed ai-config-ai-agents-allowed ai-coding-session-ses-0c -
✅ Passed ai-config-no-dangerous-commands ai-coding-session-ses-0c -
✅ Passed ai-config-no-secrets ai-coding-session-ses-0c -
✅ Passed ai-config-mcp-servers-allowed ai-coding-session-ses-0c -

Powered by Chainloop and Chainloop Trace

@migmartri migmartri force-pushed the dw/5173-1783416967 branch from 00441fb to e16b9b2 Compare July 7, 2026 09:59
@migmartri migmartri requested a review from a team July 7, 2026 15:29

@waveywaves waveywaves left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few inline questions focused on keeping the fix tight and matching existing timeout patterns.

Comment thread app/controlplane/plugins/sdk/v1/fanout.go Outdated
Comment thread app/controlplane/internal/dispatcher/dispatcher.go Outdated
Comment thread app/controlplane/internal/dispatcher/dispatcher.go Outdated

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 4 files (changes from recent commits).

Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic

Comment thread app/controlplane/internal/dispatcher/dispatcher.go
jiparis
jiparis previously approved these changes Jul 13, 2026
- Bump MaxElapsedTime from 10s to 5m (~12-15 retry attempts)
- Add per-attempt context.WithTimeout (10s) so a single hung HTTP
  request cannot consume the entire retry budget
- Set http.Client.Timeout on the webhook plugin to match the
  per-attempt deadline (shared via sdk.PerAttemptTimeout)
- Wrap backoff with backoff.WithContext so the retry loop stops
  promptly when the parent context is cancelled
- Replace deprecated io/ioutil with io in the webhook plugin

Refs chainloop-dev#3269
Closes chainloop-dev#39

Assisted-by: OpenCode
Signed-off-by: Miguel Martinez Trivino <miguel@chainloop.dev>

Chainloop-Trace-Sessions: ses_0c4100537ffeM74j5m1U4O10fe
Reflects the new per-attempt HTTP timeout behavior introduced in the
previous commit.

Assisted-by: OpenCode
Signed-off-by: Miguel Martinez Trivino <miguel@chainloop.dev>

Chainloop-Trace-Sessions: ses_0c4100537ffeM74j5m1U4O10fe
Add OpenCode to the examples and state explicitly that the value must
match the tool that produced the work, not the underlying model, and
must not be copied blindly from an example.

Assisted-by: OpenCode
Signed-off-by: Miguel Martinez Trivino <miguel@chainloop.dev>

Chainloop-Trace-Sessions: ses_0c4100537ffeM74j5m1U4O10fe
… SDK

Move the per-attempt HTTP timeout out of the plugin SDK and into the
webhook plugin as a local constant, addressing review feedback that
the control-plane retry policy should not be part of the SDK API.

- Remove sdk.PerAttemptTimeout from fanout.go
- Remove dispatcher per-attempt context.WithTimeout and backoff.WithContext
  (the production caller uses context.TODO(), making cancellation inert)
- Add local perAttemptTimeout constant in the webhook plugin
- Remove tests for the removed dispatcher-level timeout behavior

Assisted-by: OpenCode
Signed-off-by: Miguel Martinez Trivino <miguel@chainloop.dev>

Chainloop-Trace-Sessions: ses_0c4100537ffeM74j5m1U4O10fe
@migmartri migmartri merged commit d1789a9 into chainloop-dev:main Jul 13, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Improve fanout webhook delivery timeouts and retry budget

3 participants