Replies: 8 comments 7 replies
Proposal: HTTP Source Connector (Webhook Gateway)SummaryAn HTTP source connector that embeds an HTTP server inside the Source plugin, accepts incoming webhook POST requests, and produces messages to Iggy topics. This is the inverse of the HTTP sink connector (#2925) and completes the HTTP integration pair. Key features:
MotivationGap in the Connector EcosystemAll three existing source connectors are poll-based — they reach out to external systems:
Missing: A push-based source that accepts incoming HTTP requests. Webhook ingestion is the dominant pattern for receiving real-time events — SaaS integrations (GitHub, Stripe, Slack), IoT devices, CI/CD callbacks, inter-service events. Without this, every Iggy user needing webhook → topic routing must build a standalone HTTP server + Iggy producer, duplicating connection management, error handling, backpressure, and graceful shutdown logic. Architecture: Push-to-Pull BridgeThe Source trait is inherently pull-based ( Three Concurrent Tasks
SDK Compatibility AnalysisThe Source SDK was designed for poll-based sources but is fully compatible with a server-style source:
Critical constraint: Background tasks must not clone the Endpoint Registry (Event-Sourced)Why Not Just Static Config?Static
DesignThe connector maintains a live endpoint registry backed by a dedicated Iggy config topic. On startup, it replays the topic from offset 0 to reconstruct state. Then subscribes for live updates. Endpoint types:
Ephemeral endpoints use a cryptographically random 32-char hex ID. The URL itself acts as a bearer token (128 bits of entropy, same as Fluvio's webhook gateway, Slack webhook URLs). Each endpoint maps to a target topic with optional per-endpoint HMAC auth. Config topic events: {"action": "register", "endpoint_id": "a3f8c2e1...", "topic": "github_events",
"auth_type": "hmac-sha256", "hmac_header": "X-Hub-Signature-256", ...}
{"action": "revoke", "endpoint_id": "a3f8c2e1...", "reason": "compromised"}
{"action": "update", "endpoint_id": "a3f8c2e1...", "auth_secret": "new_secret"}Graceful fallback: If the config topic doesn't exist, the connector operates in static-only mode (named topics from TOML, global bearer token auth). The config topic is opt-in. Multi-Instance CoordinationAll instances consume the same config topic. Since events are ordered, all instances converge to the same registry state: Consistency model: eventual (typically <100ms propagation). Acceptable for webhook endpoints — same model used by DNS and distributed caches. Lock-Free Hot PathThe HTTP request handling path must be as fast as possible — target <100μs from TCP accept to HTTP 200. Why these data structures:
No mutex anywhere between HTTP request arrival and queue enqueue. poll() Implementationasync fn poll(&self) -> Result<ProducedMessages, Error> {
loop {
let mut batch = Vec::with_capacity(self.max_batch_size);
while let Some(msg) = self.queue.pop() {
batch.push(msg.into_produced_message());
if batch.len() >= self.max_batch_size { break; }
}
if !batch.is_empty() {
return Ok(ProducedMessages { schema: Schema::Raw, messages: batch, state: None });
}
// Wait for notification or timeout
tokio::select! {
_ = self.notify.notified() => continue,
_ = tokio::time::sleep(Duration::from_millis(100)) => {
return Ok(ProducedMessages { messages: vec![], .. });
}
_ = self.cancel_token.cancelled() => {
return Ok(ProducedMessages { messages: vec![], .. });
}
}
}
}Configurationtype = "source"
key = "http"
enabled = true
name = "webhook_gateway"
path = "target/release/libiggy_connector_http_source"
# Source connectors use StreamProducerConfig (singular `topic`).
# Multi-topic routing handled internally by the connector (see Blocker 1).
[[streams]]
stream = "webhooks"
topic = "ingest"
schema = "raw"
batch_length = 100
linger_time = "5ms"
[plugin_config]
bind_address = "0.0.0.0"
port = 9090
max_body_size_bytes = 1048576 # 1MB
buffer_capacity = 10000
max_batch_size = 500
# Config topic (optional — enables endpoint registry + hot reload)
config_topic_stream = "platform"
config_topic_name = "http-source-config"
iggy_connection_string = "tcp://127.0.0.1:8090"
# Global auth (for /topics/{name} endpoints)
auth_bearer_token = "global-webhook-secret"
# Instance identity (for multi-instance setups)
instance_id = "webhook-01"
# HTTP metadata forwarding as Iggy message headers
include_http_metadata = true
forward_headers = ["X-Request-ID", "X-Correlation-ID"]
health_check_enabled = trueRequest/Response ContractNamed TopicEphemeral EndpointError Responses
HealthHMAC ValidationPer-endpoint HMAC validation over raw request body bytes (never re-serialized JSON):
Using {"action": "register", "endpoint_id": "...", "topic": "github_events",
"auth_type": "hmac-sha256", "hmac_header": "X-Hub-Signature-256",
"hmac_prefix": "sha256=", "auth_secret": "whsec_..."}Defense in depth: URL is unguessable (128-bit) AND HMAC signature must be valid. Either can be rotated independently. Async AcknowledgmentRespond before producing to Iggy: Delivery is at-most-once from the caller's perspective. 200 = "queued locally," not "persisted to Iggy." Webhook senders implement retry-on-timeout, providing at-least-once from the caller's side. Comparison: HTTP Sink vs HTTP Source
Architectural Questions for the Team
ScopeIncluded
Not Included
RBAC: Deliberate Non-GoalWe considered and explicitly rejected building RBAC (Role-Based Access Control) into the connector. The connector performs authentication (verify the caller is who they claim) but delegates authorization (is this caller allowed to perform this action) to the layers above and below it. The connector sits between two systems that already have their own access control: Adding a third authorization layer in the middle creates configuration complexity (three places to manage permissions with no clear source of truth), consistency headaches (permissions across layers diverge), and scope creep (role definitions, permission grants, group management — an entire authorization system that would dwarf the connector). How authorization is already handled at each layer:
Key insight: Ephemeral endpoints with per-endpoint auth are functionally equivalent to per-resource authorization. Each integration gets its own endpoint with its own secret. Revoking the endpoint (via config topic) is equivalent to revoking access. No role hierarchy needed. What would change this: If Iggy develops a first-party connector authorization framework (e.g., connector-level ACLs managed by the server), the HTTP source should integrate with it rather than maintaining its own. Estimated Size~3600-4650 lines across Pre-Implementation Review ResultsWe ran 5 specialized review agents against this design before any code was written. 25 findings (7 CRITICAL, 18 HIGH). 17 fixed in the design, 4 resolved with detailed implementation notes, 3 are architectural blockers requiring team input, 1 is an inherited runtime limitation. Architectural Blockers (Need Team Input)Blocker 1: SDK Single-Topic Routing
We see four options:
We lean toward (b) for initial implementation, with (a) as a follow-up SDK enhancement. What does the team prefer? Blocker 2: Shutdown Data LossThe runtime's Options:
Would a PR to adjust the shutdown order in Blocker 3: Config Consumer FailureIf the config consumer's Iggy connection drops, the HTTP server continues with a frozen registry — revoked endpoints stay active. We've added health signaling ( Key Design Corrections From ReviewThese have already been incorporated into the design above:
Additional Type DefinitionsQueuedMessage (the hot-path bridge between HTTP handler and poll): struct QueuedMessage {
topic: String, // Target topic, already resolved
payload: Vec<u8>, // Raw HTTP body bytes
http_metadata: HashMap<String, String>, // Pre-filtered forward headers
received_at: std::time::Instant, // For queue-time latency metrics
}EndpointId (validated newtype, 128 bits of entropy): struct EndpointId(String); // Exactly 32 lowercase hex chars, validated at constructionConfigEvent (serde-tagged, compile-time field validation): #[derive(Deserialize)]
#[serde(tag = "action", rename_all = "snake_case")]
enum ConfigEvent {
Register(RegisterEndpoint),
Revoke(RevokeEndpoint),
Update(UpdateEndpoint),
Config(ConfigOverride), // Allowlisted keys with range validation
}Looking forward to your feedback on the design, especially the architectural questions above. Happy to discuss any aspect in detail. |
|
hey @mlevkov, proposal looks good. i ran multiple agents against the connectors runtime, sdk and your proposal and there are findings. i verified them all against the code. tl;dr: the design is sound, but there are runtime bugs you'd inherit that need fixing first, and a few design decisions that need adjustment before code lands. existing runtime bugsshutdown ordering drops messages
this affects all source connectors. for a webhook gateway it's especially bad -- you already returned http 200 to the caller, and webhook senders don't retry on 200. fix is straightforward: call unbounded flume channel breaks your backpressure story
needs to be FileStateProvider is not crash-safe
arc::try_unwrap race in source container close
design changes neededdrop the own-IggyClient for configi know the event-sourced config topic is elegant, but creating a separate IggyClient to consume it introduces split-brain risk. you now have two independent tcp connections with independent failure modes. if the config connection drops while the main one is healthy, revoked endpoints stay active indefinitely. that's a security hole. use the runtime's fail-closed on stale config is mandatoryif your config source is unreachable beyond a timeout, return 503. not "continue with last known good." a revoked endpoint staying active because your config consumer hiccupped is not an acceptable failure mode. document the staleness bound as an sla. delivery semantics need to be front and centerthe "async acknowledgment" section kinda buries the lede. http 200 means "queued in volatile connector memory." messages can be lost on crash, shutdown race, or oom. webhook providers (github, stripe) don't retry on 200. users need to understand this upfront, not discover it in a subsection. suggest making this a top-level section in the readme with a clear statement: "delivery guarantee: at-most-once. http 200 means accepted for processing, not durably stored." split the code into modules3600-4650 lines should be: on the three blockersblocker 1 (multi-topic routing): go with option (c) for v1single-topic-per-instance. the runtime already supports multiple plugin instances per loaded container ( zero sdk changes. zero abi breaks. works today. the limitation is real though -- the forwarding loop at for v2, we should open a proposal discussion for a don't block v1 on this. ship option (c) with documented limitations. open the proposal discussion. migrate in v2. blocker 2 (shutdown data loss): yes, fix itconfirmed the bug. the fix should be a separate pr since it affects all source connectors. happy to review. blocker 3 (config consumer failure): use ConnectorsConfigProvidercovered above. the event-sourced iggy topic approach is clever but operationally fragile. runtime's existing config mechanism + management endpoints on your axum server covers the same use cases without the split-brain risk. nice-to-haves (file separately)
summary
overall: strong proposal, solid engineering. the prerequisite runtime fixes are the main blocker. once those land, option (c) gets you a shippable v1 with clear documented limitations, and the PushSource proposal discussion charts the path to at-least-once in v2. looking forward to the pr. |
|
@hubcio Awesome! Thank you for taking time and deep insight. Very much appreciate it. I will address all of the noted priorities. |
|
Re the delivery guarantee:
I would suggest rephrasing, as the current guarantee is weaker than at-most-once. It is best-effort delivery. |
|
Solving this ^^ opens a whole set of new challenges, but perhaps we could add a TODO in the code to track this gap? |
|
@kriti-sc We could have TODO or try to address it. I will give it a thought. |
HTTP Source: refreshed v1 plan (grounded against master, 2026-07-16)Picking this back up after three months. I re-verified every item from @hubcio's review against current master before writing a line of code. Status first, then the v1 shape we're committing to. Prerequisite status
v1 plan: option (c), as recommendedAdopting single-topic-per-instance with a shared listener:
Config and dynamic endpoints: your suggestion, plus one findingOwn-IggyClient config consumer is dropped, as you recommended. Static endpoints (named topic path, secret-path endpoints, auth) live in TOML through the normal For real-time operations (revoking a compromised endpoint, provisioning endpoints programmatically) we're adopting your management-endpoints suggestion: The finding worth calling out: dynamic registrations can be made durable with zero new runtime machinery via
So revocations survive restarts, nothing resurrects, and there's no second connection to go stale. The fail-closed-on-staleness concern from the original design doesn't apply: the registry is process-local and mutations are synchronous. The known limits are documented: persistence lands on the next successful poll cycle (a Other updates baked into the plan
Sequencing
Given #3580 (connectors repo split) lists "land the open connector PRs" as a pre-split step, I'd like to land 1 and 2 inside that window; the v2 trait work gets more expensive once Two things to confirm before I start:
If no objections, I'll start with the prerequisite PR. |
|
Full refreshed v1 design attached for reference — covers the shared-server lifecycle rules, endpoint state machine, ConnectorState persistence details, module layout, and the blocker resolution log. |
|
hey @mlevkov, iverified the refresh against master just now, it looks ok and valid. your questions:
one change request for the prerequisite PR: use crossfire, not flume. it's already a workspace dep (3.1.19, current release), it's what shard and server-ng standardized on, and the exact shape exists in two things the bounded PR must handle: 1) the close-wedge - iggy hangs, channel fills, callback blocks in send, and nits: enumerate the duplicate window too (sender retries are why this isn't at-most-once), and |
|
The bounded-channel prerequisite PR is up: #3795. One correction to the spec discovered during review: crossfire's blocking sender does have |
|
Filed #3796 for the SDK-side follow-up: handing off the plugin-runtime worker ( |
|
A corresponding PR is issued -> #3797 |
v1 is implemented and open as #3798CI is green: 99 checks pass, 1 skipped. It shipped as the shape approved in this thread, with no architectural departures: single topic per instance, one shared axum listener across instances, management API on a separate admin listener (loopback + token + off by default), dynamic endpoints persisted through Also up since my last comment: #3797, the SDK worker handoff for #3796. Corrections to the design as postedBuilding it turned up places where the design attached to this thread is wrong in detail. The shape held. These are the details, and each one changes what a reader of that document should expect. The Stripe example in the Configuration section could not have worked. It shows
State must ride an empty batch. The runtime saves plugin state only on the success branch of the Iggy send. Attaching state to a non-empty batch means an unrelated send failure silently drops the mutation, losing a revocation tombstone after the plugin has already marked it submitted. An empty send always returns The config structs derive no Two smaller ones. The design's 503 for "instance closing" is a 404 in practice, because deregistration is atomic under the listener registry lock and no observable window exists where a route resolves to a closing instance (503 survives on Two upstream defects found on the wayBoth were found while building this and are filed separately, since neither is specific to this connector:
@hubcio the connector is ready for review whenever you have time. #3795 and #3797 were both behind master and are now rebased onto it, with CI re-running. |
|
Status follow-up to the comment above, now that everything has settled. All five PRs are CI-green against current master:
#3795 and #3797 were behind master and are now rebased onto it. #3798 and #3804 each picked up a test commit since the last comment. The one worth mentioning here is on #3798, because it is a review-process point rather than a connector one: the management API's auth guard turned out to be untested on all four mutating routes. Coverage regions are per call site, so #3802 is deliberately still open. #3804 covers the startup warning and documents the control API as privileged, but redacting the configuration responses needs a maintainer decision: @hubcio all five are ready for review whenever you have time. |
Suggested review orderAsked for a sequence, so here is one with the reasoning, plus the two places where the review needs a decision from you rather than an approval. First, what does not constrain the order: I simulated all ten pairwise merges with
The one hard constraint#3795 before #3797 and #3798, for comprehension rather than merge order:
Everything else is preference. Two things that need a decision, not an approval
If you would rather front-load the decisions#3795 → #3804 → #3803 → #3797 → #3798 puts both judgment calls in the first two slots and leaves the three mechanical reviews after them. |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Hi team,
Following up on the HTTP sink connector (#2925), we'd like to propose its inverse — an HTTP source connector that acts as a webhook gateway. It embeds an HTTP server inside the Source plugin, accepts incoming POST requests, and produces messages to Iggy topics.
The connector ecosystem has 3 sources today, all poll-based (random, PostgreSQL, Elasticsearch). There's no push-based source for accepting inbound webhooks — the most common pattern for receiving real-time events from SaaS integrations (GitHub, Stripe, Slack), IoT devices, CI/CD pipelines, and inter-service communication.
Pre-Implementation Review
We ran 5 specialized review agents against this design before writing any code. 25 findings (7 CRITICAL, 18 HIGH). 21 have been corrected in the design. 3 are architectural blockers that require team input:
Blocker 1: SDK Single-Topic Routing
ProducedMessagehas no topic field. The runtime creates oneIggyProducerfor the last[[streams]]entry. Multi-topic routing (different webhook endpoints → different topics) is the connector's primary value proposition, but it cannot be expressed through the current SDK contract.We lean toward using the connector's own
IggyClientto produce directly (bypassing the forwarding loop), with an SDK extension as a follow-up. What does the team prefer?Blocker 2: Shutdown Data Loss
SourceManager::stop_connectorcallscleanup_sender(removes the flume channel) BEFORE callingiggy_source_close. Messages drained bypoll()during shutdown hit a dead callback. Would a PR to adjust the shutdown order be welcome? (This affects all source connectors, not just HTTP.)Blocker 3: Config Consumer Failure
The connector creates its own
IggyClientto consume a config topic (event-sourced endpoint registry). If that connection drops, the HTTP server continues with a frozen registry — revoked endpoints stay active. We've added health signaling (degraded status, optional rejection). Is this failure mode acceptable?Design Highlights
crossbeam::queue::ArrayQueue) →poll()drains → Iggy/e/{random_id}) — individually revocable, with optional per-endpoint HMAC validationArcSwapfor registry,ArrayQueue+Notifyfor buffer — no mutex between TCP accept and HTTP 200CancellationTokentreeFull design document with all corrections, type definitions, and review findings is in the comment below.
Looking forward to your feedback, especially on the three blockers.
All reactions