[DISCUSS] proposal: move connectors into a dedicated iggy-connectors repository

[hubcio]

I want to move the whole connectors subsystem (core/connectors/: runtime, sdk,
all sinks and sources + tests) out into its own apache/iggy-connectors repo. nothing
decided yet, just want to hear what people think.

the reason is CI time. our test scoping walks the workspace dependency graph, and any
change to iggy-server ends up pulling the whole rust workspace in, so every
connector gets built and tested too. the plugins are cdylib crates that ci
always builds, so even a one-line server change rebuilds every connector .so
and spins up the docker/testcontainers backends. we have ~16 connector crates
now (10 sinks, 4 sources), some pulling huge trees like arrow/parquet, deltalake
and cloud sdks, mongo, elasticsearch. so people working on the server keep paying
for connectors they never touched, and it gets worse every time someone adds
another heavy one.

splitting is safe because the runtime only talks to the server as a normal
client, over iggy and iggy_binary_protocol, never server internals. testing
connectors against a released server is the same thing real users do. the only
internal contract is the ffi plugin abi in iggy_connector_sdk, and that stays
in the new repo next to the plugins that depend on it.

what we'd get: server CI stops building and container-testing connectors, the
heavy deps leave the main build, and connectors can release on their own clock.
what we'd have to deal with: server changes that break a connector get caught
later (on a release or nightly, not in the pr); iggy_connector_sdk becomes a
real published crate with an abi/semver policy to keep; the crates connectors
depend on (iggy, iggy_common, iggy_binary_protocol, configs) need to be
published and pinned; and since we're incubating, a new repo means going through
asf infra.

rough order if we do it: land the open connector prs, file an apache infra
ticket to create the apache/iggy-connectors repo, split with history into it,
publish and pin the dep crates, set up ci that runs against the latest released
server, then drop core/connectors/ from the main workspace.

a few of us already like the idea, but i'd rather argue it out here first. if
you maintain or run a connector, or you think this is a bad move, say so.

[kparisa]

+1 (Apache Iggy PPMC)

Also, I think we should also create an iggy-example-apps (or just iggy-examples, but there is /examples folder at the moment) repository where the community can contribute real-world use cases and example implementations.

What are everyone’s thoughts on the current /examples folder? currently, they are focused on SDK usage across different languages. Should we keep those SDK-level examples in the main repo and move broader iggy-example repository?

[mlevkov]

I have one already ;) . I have several update notes for that one -> https://github.com/mlevkov/iggy_sample , given the latest releases since creation of the project.

[numinnex]

+1 binding (Apache Iggy PPMC)

I think moving connectors to separate repo would happen sooner or later (number of supported connectors started to grow really fast). Regarding examples, I think those examples that we currently have should stay local to where SDK lives (they are very simple demonstration of basic functionalities), but I don't mind creating a dedicated repo with more detailed applications (we already have couple of those community made, those could be contributed to the new repo).

[ryerraguntla]

+1 for separate connectors.
Couple of questions.

1. Apart from naming convention, is there a way to group these repositories as related and dependent?
2. With separate repositories, How are the milestone releases are managed and published for the complete Product?

[hubcio]

1. we will treat it as dependant via cargo dependencies
2. we will track everything in the connectors repository. if you think about it, connectors are independent of iggy itself. those are just extensions built on top.

[ryankert01]

+1 (non binding)

[mmodzelewski]

+1 I think we would only benefit from separating the connectors. It's a clean cut, so it should be fairly straightforward to move.

[mlevkov]

Neutral.

If you do decide to split, would have to make sure that cross repo references are maintained in strict alliance to avoid accidental stale state. I'm confident that such would be the case anyway.

[hubcio]

worry not. the other repository will be first class citizen, same CI, same guarantees and approach to quality.

[mlevkov]

Then
+1

Just make sure the documentation explicitly references the related repo for discoverability.

[spetz]

+1 it'll be much easier to maintain the core server + SDK vs connectors (which have been significantly growing for a while now).

[mlevkov]

Sounds good. Got my vote +1.

[kriti-sc]

I upvoted earlier but have a followup q:

the connectors will keep growing. wouldn't the other repo eventually run into the same issue?
Is there no other workaround re the CI situation, for ex. not running connector tests if connector + adjacent has not changed?

also wondering if there are any other potential workarounds for the CI situation? For ex., could we skip running connector tests when connector + adjacent code has not changed?

[hubcio]

@kriti-sc no worries. we'll create a port of DAG-based mechanism that would run all connectors tests ONLY if connectors SDK has changed (or Cargo.lock/toml/CI infra). otherwise, only respective connectors would be running.

[kriti-sc]

Could that be implemented in this repo itself?

[hubcio]

it's already implemented in this repo. it's just that if you change something in SDK, all connectors are getting retested due to DAG dependency detected via cargo rail + integration crates is a bundle of all integrations tests (connectors, server, data integrity, etc)

[kriti-sc]

The optics dimension should also be considered while making this decision. Contributing to apache/iggy carries a visibility which the apache/iggy-connectors repo won't. This might harm contributions. So it would be important to figure out how to signal that connectors are core, not peripheral.

[hubcio]

we're fully aware of that - we for sure gonna setup the discord notifications, emailing lists and promote the new repository in social media. tbh i think it can get bigger than iggy core in terms of contributions per time frame once more people will be aware that it exists.