refactor(downstream): source features from philanthropy.ingest package

Drop the local unischema_features.py shim (and its Python contract tests) in
favor of importing read_constituent_events / constituent_events_to_features
directly from the PhilanthroPy package, the single source of truth for the
feature contract. Update pipelines, docs, and dbt notes to the new column
names (gift_count, event_attendance_count, total_gift_amount,
distinct_source_systems).

- downstream-ml.yml: replace the deleted-pytest gate with a dependency-free
  py_compile check; repoint the optional smoke to philanthropy.ingest
- repositoryGuard: skip .venv/venv/__pycache__ in the secret scan

Author: shivamlalakiya

.github/workflows/downstream-ml.yml

@@ -13,8 +13,8 @@ on:
       - '.github/workflows/downstream-ml.yml'
 
 jobs:
-  feature-contract:
-    name: Python feature contract tests
+  pipeline-compile:
+    name: Downstream pipeline scripts compile
     runs-on: ubuntu-latest
 
     steps:
@@ -26,20 +26,19 @@ jobs:
         with:
           python-version: '3.11'
 
-      - name: Install dependencies
-        run: |
-          pip install pandas pytest
-          pip install -r examples/downstream/requirements-philanthropy.txt || pip install pandas pytest
-
-      - name: Run feature contract tests
+      # The feature contract now lives in philanthropy.ingest (PhilanthroPy repo)
+      # and is pinned TS-side by tests/unit/schemaContract.test.ts, so there is no
+      # local Python contract test. This gate just catches syntax breakage in the
+      # example scripts (py_compile parses only — no third-party imports executed).
+      - name: Byte-compile downstream pipeline scripts
         working-directory: examples/downstream
-        run: pytest tests/test_feature_contract.py -q
+        run: python3 -m py_compile *.py
 
   philanthropy-smoke:
     name: PhilanthroPy pipeline smoke (optional)
     runs-on: ubuntu-latest
     needs:
-      - feature-contract
+      - pipeline-compile
     continue-on-error: true
 
     steps:
@@ -54,6 +53,6 @@ jobs:
       - name: Install PhilanthroPy stack
         run: pip install -r examples/downstream/requirements-philanthropy.txt
 
-      - name: Smoke test unischema_features import
+      - name: Smoke test philanthropy.ingest import
         working-directory: examples/downstream
-        run: python3 -c "from unischema_features import REQUIRED_FEATURE_COLUMNS; print(len(REQUIRED_FEATURE_COLUMNS), 'columns')"
+        run: python3 -c "from philanthropy.ingest import constituent_events_to_features, read_constituent_events; print('philanthropy.ingest OK')"

docs/downstream-pipeline.md

@@ -8,7 +8,7 @@ End-to-end path from UniSchema pilot to warehouse analytics and ML with [Philant
 flowchart LR
   Vendors[Vendor webhooks] --> UniSchema[UniSchema API]
   UniSchema --> Egress[Local or S3 egress]
-  Egress --> Features[unischema_features.py]
+  Egress --> Features[philanthropy.ingest]
   Egress --> dbt[dbt staging and marts]
   Features --> PhilanthroPy[PhilanthroPy pipelines]
   dbt --> PhilanthroPy
@@ -104,7 +104,6 @@ events group onto one key across vendors before RFM aggregation.
 |--------|--------|
 | `philanthropy_crm_pipeline.py` | **Primary** — CRM labels + `DonorPropensityModel` |
 | `philanthropy_pipeline.py` | Egress-only demo with proxy labels |
-| `unischema_features.py` | Feature table library |
 | `crm_join_example.py` | CRM join (`externalConstituentId` or email) |
 
 CRM join prefers `externalConstituentId` → CRM `constituent_id`, then email fallback.

docs/philanthropy-integration.md

@@ -6,12 +6,14 @@
 
 ```
 ConstituentEvent (egress JSON / S3 NDJSON)
-    → unischema_features.py (per-email feature table)
+    → philanthropy.ingest.constituent_events_to_features (per-donor feature table)
     → optional CRM join (engagement_tier, lifetime_giving)
     → PhilanthroPy DonorPropensityModel / LapsePredictor
     → affinity or lapse scores (0–100)
 ```
 
+Both reading and aggregation live in PhilanthroPy's `philanthropy.ingest` — the single source of truth. `read_constituent_events` walks UniSchema's date-partitioned egress tree (recursively; skips `.manifest.json` sidecars); `constituent_events_to_features` is **leakage-safe** (recency anchored to the batch, never a moving "now") and **deduplicates** redelivered webhooks by `eventId`.
+
 Warehouse path: dbt `mart_constituent_rfm_features` → export CSV → same PhilanthroPy pipeline.
 
 ## Install
@@ -38,31 +40,29 @@ npm run downstream-demo
 
 ## Feature column contract
 
-`examples/downstream/unischema_features.py` aggregates events to one row per `constituent_email`:
+`philanthropy.ingest.constituent_events_to_features` aggregates events to one row per donor, indexed by `constituent_id` (the `externalConstituentId` when present, else `constituentEmail`). This contract is owned and tested in the PhilanthroPy repo (`tests/test_ingest.py`):
 
 | Column | Type | Source |
 |--------|------|--------|
-| `constituent_email` | string | Group key |
-| `donation_count` | int | `eventType == DONATION` |
-| `registration_count` | int | `eventType == EVENT_REGISTRATION` |
-| `email_click_count` | int | `eventType == EMAIL_CLICK` |
-| `total_amount` | float | Sum of donation `amount` |
-| `total_gift_amount` | float | Alias of `total_amount` (PhilanthroPy naming) |
-| `event_attendance_count` | int | Alias of `registration_count` |
-| `last_donation_date` | ISO string | Latest donation `createdAt` |
-| `last_event_date` | ISO string | Latest any-event `createdAt` |
-| `first_event_date` | ISO string | Earliest any-event `createdAt` |
-| `years_active` | float | Years from `first_event_date` to reference date |
-| `metadata_key_count` | float | Sum of `normalizedMetadata` key counts |
-
-Contract tests: `examples/downstream/tests/test_feature_contract.py`
+| `constituent_email` | string | First email seen for the donor |
+| `total_gift_amount` | float | Sum of `DONATION` `amount` (single-currency; warns on mixed) |
+| `gift_count` | int | Count of `DONATION` events |
+| `event_attendance_count` | int | Count of `EVENT_REGISTRATION` events |
+| `email_click_count` | int | Count of `EMAIL_CLICK` events |
+| `first_gift_date` | datetime | Earliest donation `createdAt` |
+| `last_gift_date` | datetime | Latest donation `createdAt` |
+| `years_active` | float | First event → reference date |
+| `recency_days` | int | Last event → reference date |
+| `distinct_source_systems` | int | Channel breadth |
+
+Index: `constituent_id` (`externalConstituentId` ?? `constituentEmail`).
 
 ### PhilanthroPy model inputs
 
 | PhilanthroPy component | UniSchema-derived columns |
 |------------------------|---------------------------|
 | `DonorPropensityModel` | `total_gift_amount`, `years_active`, `event_attendance_count` |
-| `RFMTransformer` | Use `last_donation_date`, `donation_count`, `total_amount` after date parsing |
+| `RFMTransformer` | `recency_days`, `gift_count`, `total_gift_amount` |
 | `WealthScreeningImputer` | CRM `lifetime_giving` → `estimated_net_worth` |
 | `LapsePredictor` | CRM-derived lapse labels + RFM features from warehouse |
 
@@ -86,7 +86,6 @@ Example CRM columns (see `samples/crm-golden-record.csv`):
 
 | Script | Purpose |
 |--------|---------|
-| `unischema_features.py` | Library — egress → DataFrame |
 | `philanthropy_pipeline.py` | Egress only, proxy labels (demo) |
 | `philanthropy_crm_pipeline.py` | **Recommended** — CRM labels + `DonorPropensityModel` |
 
@@ -101,11 +100,10 @@ Example CRM columns (see `samples/crm-golden-record.csv`):
 ## sklearn pipeline example
 
 ```python
-from pathlib import Path
+from philanthropy.ingest import constituent_events_to_features, read_constituent_events
 from philanthropy.models import DonorPropensityModel
-from unischema_features import egress_dir_to_features
 
-features = egress_dir_to_features(Path("data/egress"))
+features = constituent_events_to_features(read_constituent_events("data/egress"))
 X = features[["total_gift_amount", "years_active", "event_attendance_count"]].to_numpy()
 
 # y from CRM engagement_tier after join — see philanthropy_crm_pipeline.py

docs/schema-governance.md

@@ -65,7 +65,7 @@ Add `VOLUNTEER_SHIFT` to `EventTypeSchema` in `src/schema/master.ts`.
 |-----------|--------|
 | dbt `mart_constituent_engagement_daily` | Add `volunteer_shift_count` column |
 | dbt `mart_constituent_rfm_features` | Optional engagement feature |
-| `unischema_features.py` | Add `volunteer_shift_count` to contract |
+| `philanthropy.ingest` (PhilanthroPy repo) | Add `volunteer_shift_count` to the feature contract |
 | PhilanthroPy pipelines | Map to engagement features or separate model |
 
 ### Do not implement until

examples/downstream/README.md

@@ -33,7 +33,7 @@ npm run downstream-demo
 
 ## Feature column contract
 
-`unischema_features.py` aggregates egress JSON to a per-constituent DataFrame. Required columns are tested in `tests/test_feature_contract.py`.
+`philanthropy.ingest` is the single source of truth: `read_constituent_events` walks the date-partitioned local egress tree (recursive; skips `.manifest.json`), and `constituent_events_to_features` aggregates it (leakage-safe, dedup by `eventId`). Owned and tested in the PhilanthroPy repo.
 
 | Column | PhilanthroPy use |
 |--------|------------------|
@@ -47,7 +47,6 @@ Full contract → [philanthropy-integration.md](../../docs/philanthropy-integrat
 
 | Script | Purpose |
 |--------|---------|
-| `unischema_features.py` | Egress → feature DataFrame |
 | `philanthropy_crm_pipeline.py` | **Recommended** ML path with CRM labels |
 | `philanthropy_pipeline.py` | Demo with proxy labels |
 | `crm_join_example.py` | CRM join (`externalConstituentId` or email) |

examples/downstream/dbt/README.md

@@ -33,7 +33,7 @@ dbt run
 Export `mart_constituent_rfm_features` to CSV or query via Python:
 
 ```python
-# Warehouse export columns align with unischema_features.py contract
+# Warehouse mart produces model-ready aliases (total_gift_amount, event_attendance_count)
 # See docs/philanthropy-integration.md
 ```
 

examples/downstream/dbt/models/marts/schema.yml

@@ -13,9 +13,10 @@ models:
 
   - name: mart_constituent_rfm_features
     description: >
-      Per-constituent RFM-style rollup for PhilanthroPy batch scoring.
-      Column contract matches examples/downstream/unischema_features.py
-      and docs/philanthropy-integration.md.
+      Per-constituent RFM-style rollup for PhilanthroPy batch scoring. A
+      warehouse-side alternative to the philanthropy.ingest event aggregation:
+      it produces model-ready aliases (total_gift_amount, event_attendance_count)
+      consumed directly by PhilanthroPy models. See docs/philanthropy-integration.md.
     columns:
       - name: constituent_email
         tests: