Skip to content

fix(connectors): hand off SDK worker during blocking send callbacks - #3797

Open
mlevkov wants to merge 1 commit into
apache:masterfrom
mlevkov:sdk-callback-worker-handoff
Open

fix(connectors): hand off SDK worker during blocking send callbacks#3797
mlevkov wants to merge 1 commit into
apache:masterfrom
mlevkov:sdk-callback-worker-handoff

Conversation

@mlevkov

@mlevkov mlevkov commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

Summary

Closes #3796. The send callback runs synchronously inside the SDK's polling task (handle_messages in core/connectors/sdk/src/source.rs), on the tokio runtime shared by every connector instance loaded from the same plugin library (static RUNTIME: OnceLock<Runtime> in lib.rs). A callback that blocks pins one worker for the duration. With #3795's bounded forwarding channel, a callback legitimately blocks for backpressure while the channel is full, so enough saturated instances of one library can occupy all workers, and iggy_source_close for a sibling instance then waits behind them: the close blocks on the sibling's polling task getting scheduled to observe its shutdown signal.

Change

One call site: wrap the callback invocation in tokio::task::block_in_place, so the worker is handed off before the callback runs and the runtime keeps scheduling sibling tasks regardless of how long the callback blocks.

Notes from the analysis in #3796:

  • The wrap has to live in the SDK, not the runtime: block_in_place consults the calling thread's tokio context, which is only set on the plugin runtime's own worker threads.
  • The SDK runtime is Runtime::new(), i.e. multi-threaded, which block_in_place requires.
  • No FFI or ABI change. Existing plugin binaries keep working and pick the fix up when rebuilt against the updated SDK.

This composes with #3795 (bounded channel + shutdown signaling) but does not depend on it: any long-blocking callback benefits.

Test plan

  • cargo clippy -p iggy_connector_sdk --all-targets --all-features -- -D warnings clean
  • cargo test -p iggy_connector_sdk --all-features passes
  • cargo build -p iggy_connector_stdout_sink -p iggy_connector_random_source -p iggy-connectors (macro consumers + runtime rebuild cleanly)
  • Behavior under saturation is exercised end to end by the integration suite in CI

@github-actions

github-actions Bot commented Aug 2, 2026

Copy link
Copy Markdown

Thanks for the PR. It is labeled S-waiting-on-review and queued for review.

Slash commands (own line, regular comment) move it around the queue:

  • /ready - back to S-waiting-on-review after addressing feedback
  • /author - flip to S-waiting-on-author while you finish changes
  • /request-review @user-or-team - request a reviewer

See CONTRIBUTING.md for details.

@github-actions github-actions Bot added the S-waiting-on-review PR is waiting on a reviewer label Aug 2, 2026
@codecov

codecov Bot commented Aug 2, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 51.07%. Comparing base (1453114) to head (2772550).
⚠️ Report is 19 commits behind head on master.

Additional details and impacted files
@@              Coverage Diff              @@
##             master    #3797       +/-   ##
=============================================
- Coverage     75.72%   51.07%   -24.66%     
  Complexity      969      969               
=============================================
  Files          1322     1321        -1     
  Lines        159363   139283    -20080     
  Branches     132746   112666    -20080     
=============================================
- Hits         120684    71136    -49548     
- Misses        35041    65177    +30136     
+ Partials       3638     2970      -668     
Components Coverage Δ
Rust Core 44.79% <100.00%> (-30.90%) ⬇️
Java SDK 62.71% <ø> (ø)
C# SDK 72.29% <ø> (ø)
Python SDK 93.10% <ø> (ø)
PHP SDK 84.52% <ø> (ø)
Node SDK 95.23% <ø> (ø)
Go SDK 43.08% <ø> (ø)
Files with missing lines Coverage Δ
core/connectors/sdk/src/source.rs 83.89% <100.00%> (+0.21%) ⬆️

... and 365 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

github-actions Bot commented Aug 2, 2026

Copy link
Copy Markdown

/request-review: could not request mlevkov - a handle is not a repo collaborator, or the team is unknown.

@mlevkov

mlevkov commented Aug 2, 2026

Copy link
Copy Markdown
Contributor Author

/request-review @hubcio

@github-actions
github-actions Bot requested a review from hubcio August 2, 2026 03:00
The send callback runs synchronously inside the SDK's polling task, on
a tokio runtime shared by every connector instance loaded from the same
plugin library. A callback that blocks for backpressure pins one worker
for the duration, and with enough saturated instances iggy_source_close
for a sibling waits behind them, since the close blocks on the sibling's
polling task getting scheduled to observe its shutdown signal.

Wrap the callback in tokio::task::block_in_place so the worker is
handed off before the callback runs. The SDK runtime is multi-threaded,
which block_in_place requires. No FFI or ABI change; plugins pick this
up when rebuilt against the updated SDK.

Fixes apache#3796.

Co-authored-by: Claude <noreply@anthropic.com>
@mlevkov
mlevkov force-pushed the sdk-callback-worker-handoff branch from c73a8f5 to 2772550 Compare August 2, 2026 21:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

S-waiting-on-review PR is waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SDK: hand off the plugin-runtime worker during blocking source send callbacks

1 participant