Skip to content

feat(connectors): add moss-connector-zeroentropy#490

Open
HarshaNalluru wants to merge 2 commits into
mainfrom
feat/moss-connector-zeroentropy
Open

feat(connectors): add moss-connector-zeroentropy#490
HarshaNalluru wants to merge 2 commits into
mainfrom
feat/moss-connector-zeroentropy

Conversation

@HarshaNalluru

@HarshaNalluru HarshaNalluru commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

What

A new data connector, moss-connector-zeroentropy, that migrates a ZeroEntropy collection into a Moss index. Follows the established moss-data-connector/_template pattern exactly (same shape as the Supabase/MongoDB/DynamoDB connectors).

from moss_connector_zeroentropy import ZeroEntropyConnector, ingest

source = ZeroEntropyConnector(collection_name="my_collection", api_key="...")  # or ZEROENTROPY_API_KEY
await ingest(source, project_id="...", project_key="...", index_name="my_collection")

How it works

  • ZeroEntropyConnector.__iter__ walks documents.get_info_list (an auto-paginating cursor) and yields one moss.DocumentInfo per document via a mapper.
  • Default mapping (path -> id, parsed content -> text, metadata -> metadata) means a whole collection migrates with no mapper; pass mapper= to customize.
  • The list endpoint returns metadata only, so content is fetched per document via documents.get_info(..., include_content=True). Documents that never parsed (no content) are skipped rather than indexed empty.
  • ZeroEntropy metadata is typed Dict[str, str | list[str]]; the connector coerces list values to strings (Moss requires Dict[str, str]).
  • path_prefix= scopes the migration; include_content=False migrates metadata only.

Testing

  • tests/test_zeroentropy.py — 9 mocked unit tests, all passing (default/custom mapper, skip-empty-content, include_content=False, path_prefix, auto_id, empty collection, metadata coercion).
  • tests/test_integration_zeroentropy_moss.py — skippable live round trip (ZeroEntropy → Moss → query → delete), gated on ZEROENTROPY_API_KEY / ZEROENTROPY_TEST_COLLECTION / MOSS_PROJECT_ID / MOSS_PROJECT_KEY.
  • Verified against the installed zeroentropy 0.1.0a11 SDK by introspection: method names, keyword params (collection_name, path, include_content, path_prefix), the SyncGetDocumentInfoListCursor return, and every response field the connector reads all match.

Also

  • publish-moss-connector-zeroentropy.yml (mirrors the other connectors' publish workflows).
  • Row added to packages/moss-data-connector/README.md.

CI is green (python-lint + python-sdk-test pass under the pinned ruff from #480).

Migrate a ZeroEntropy collection into a Moss index, following the
moss-data-connector _template pattern: a ZeroEntropyConnector whose __iter__
pages through documents.get_info_list and yields DocumentInfo, plus the shared
ingest() helper.

- Default mapping (path -> id, content -> text, metadata -> metadata) so a whole
  collection migrates with no mapper; overridable via mapper=.
- Fetches parsed text per document via get_info(include_content=True) and skips
  documents that never parsed (no content) rather than indexing empties.
- Coerces ZeroEntropy's str|list[str] metadata to Moss's str-only values.
- Unit tests (mocked, 9 passing) + skippable live integration test.
- Publish workflow, package README, and a row in the connectors table.

Verified against the installed zeroentropy 0.1.0a11 SDK: method names, params,
and response fields all match.
@HarshaNalluru
HarshaNalluru requested a review from r4ghu as a code owner July 25, 2026 01:35
Copilot AI review requested due to automatic review settings July 25, 2026 01:35

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new Python data connector package under packages/moss-data-connector/ to migrate a ZeroEntropy collection into a Moss index, consistent with the existing connector template and publish workflow patterns used elsewhere in the repo.

Changes:

  • Introduces moss-connector-zeroentropy package with ZeroEntropyConnector + ingest() helpers for collection migration.
  • Adds mocked unit tests and a skippable live integration test (ZeroEntropy → Moss → query → delete).
  • Wires the connector into the mono-repo docs and adds a dedicated publish workflow.

Reviewed changes

Copilot reviewed 9 out of 10 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
packages/moss-data-connector/README.md Adds moss-connector-zeroentropy to the connectors list/table.
packages/moss-data-connector/moss-connector-zeroentropy/uv.lock Adds a uv lockfile for the new connector’s dev/runtime deps.
packages/moss-data-connector/moss-connector-zeroentropy/tests/test_zeroentropy.py Mocked unit tests covering default/custom mapping, skipping empty content, prefix, auto-id, metadata coercion, etc.
packages/moss-data-connector/moss-connector-zeroentropy/tests/test_integration_zeroentropy_moss.py Skippable live end-to-end migration + query + cleanup test.
packages/moss-data-connector/moss-connector-zeroentropy/src/ingest.py Implements ingest() to create a Moss index from an iterable of DocumentInfo.
packages/moss-data-connector/moss-connector-zeroentropy/src/connector.py Implements ZeroEntropyConnector, default mapping, and metadata coercion.
packages/moss-data-connector/moss-connector-zeroentropy/src/init.py Re-exports the public connector API (ZeroEntropyConnector, ingest, etc.).
packages/moss-data-connector/moss-connector-zeroentropy/README.md Package-level usage docs and migration guidance.
packages/moss-data-connector/moss-connector-zeroentropy/pyproject.toml Declares the new package metadata, deps, and tool config.
.github/workflows/publish-moss-connector-zeroentropy.yml Adds a publish workflow mirroring existing connector publish workflows.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@github-actions

github-actions Bot commented Jul 25, 2026

Copy link
Copy Markdown

Codex review

The connector implementation is mostly coherent, but the release workflow has a release-integrity hazard around version reuse.

- Copilot: reject include_content=False with the default mapper (it reads
  'content', so it would silently index empty documents). Fail fast in __init__
  with a clear error; added a unit test.
- Codex (blocking): the publish step ran inside one matrix leg, so it could
  upload+tag before other Python versions finished. Split into a build-test
  matrix + a separate publish job that needs the whole matrix.
Comment thread .github/workflows/publish-moss-connector-zeroentropy.yml Outdated
TWINE_USERNAME: __token__
TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}
run: |
twine upload --skip-existing packages/moss-data-connector/moss-connector-zeroentropy/dist/*

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BLOCKING ```yaml
twine upload --skip-existing packages/moss-data-connector/moss-connector-zeroentropy/dist/*

`--skip-existing` makes a reused version look like a successful publish, so this job can still tag the current commit even when PyPI already contains artifacts built from a different commit. Remove `--skip-existing` and fail on existing versions, or pre-check the tag/PyPI version and only skip when you can verify the existing artifacts correspond to this exact commit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants