fix: restore Arrow download path, decouple delete list, detect stale step-01

[Burdantes]

Three fixes surfaced by the 2025-08 backfill, which failed in Phase D last night.

1. Missing google-cloud-bigquery-storage → tomography OOM

google-cloud-bigquery-storage was neither declared in pyproject.toml nor installed in the image, so every tomography download logged

Storage Read API unavailable (cannot import name 'bigquery_storage' ...); using REST download

and built result frames from JSON rows instead of Arrow buffers.

On 2025-08-01 that meant 13.7M node-edge rows in 19.5 minutes, and the retained frame plus Phase 4's allocations OOM-killed the worker at 22.1 GB inside a 28 GB cgroup, failing the whole chunk:

Memory cgroup out of memory: Killed process 1585062 (python3.14)
  total-vm:28387040kB, anon-rss:22138616kB

2026-05-27 survived the identical path at 11.2M rows, so the pipeline has been running just under this ceiling for a while. This affects every nightly, not just backfills — ~20 min of each Phase D is REST download.

Also pins pyarrow explicitly (it was only in dev extras and arriving transitively).

2. --delete-first could not clear the Phase-D correlation tables

SQL_FILES and OUTPUT_TABLES were zipped positionally, forcing one list to serve two purposes:

job	needs
SQL_FILE_TO_OUTPUT_TABLE resume map	exactly one table per SQL file, in order
delete_dates() delete list	every table the pipeline writes

The three correlation tables are written by Phase-D Python with no 1:1 SQL file, so adding them raised on strict=True. Consequence: every re-run needed a manual pre-delete or it appended a second copy of that date's rows (they use insert_rows_json, which has no self-delete).

Split into an explicit SQL_FILE_TO_OUTPUT_TABLE dict, DERIVED_OUTPUT_TABLES, and DELETE_TABLES, with an import-time check replacing strict=True.

giga_meter_measurements is deliberately excluded — it accumulates history under first-writer-wins and part of it was recovered by hand, so a delete not followed by a successful 04 loses traces permanently. A test pins that exclusion so adding it has to be a conscious decision.

3. Nothing detected a stunted merged_download_upload

Step 01 can write a partition that is present and non-empty — invisible to step_already_done() — while holding a fraction of the available measurements. Everything downstream scales down with it.

Three such dates were found by hand this week; nothing in the pipeline saw them:

date	captured	of source
2026-05-27	650,293	8.1%
2026-07-13	385,526	4.5%
2026-07-17	1,177,571	13.7%

find_stale_merged_partitions() compares each partition to its neighbour median, excluding itself so a stale date cannot drag down its own baseline. Reads INFORMATION_SCHEMA.PARTITIONS only — free — so it runs after Phase A on every invocation. Warns rather than fails, because from metadata alone a genuine traffic dip is indistinguishable from under-capture; the log line says how to confirm.

Validated against live metadata:

• replaying the three pre-repair counts flags all three (ratios 0.144 / 0.084 / 0.241 against a 0.5 cutoff)
• zero false positives across ~350 current partitions

Testing

55 passed, 2 skipped on Python 3.14, including 14 new tests across test_output_table_wiring.py and test_stale_merged_check.py. ruff check and ruff format --check clean.

🤖 Generated with Claude Code