feat: add Kalshi as a market source

[pythoryn]

Summary

Adds Kalshi as a ForecastBench market source, following the structure and conventions used by the existing market sources.

The implementation:

• Discovers active Kalshi markets and retains authoritative event and series metadata.
• Applies category balancing and the configured question cap.
• Produces canonical Kalshi market URLs.
• Preserves Kalshi's full primary and secondary resolution rules.
• Disambiguates repeated sibling question titles using the relevant Yes outcome label.
• Uses structured Kalshi timing fields to determine conservative curation cutoffs.
• Builds daily resolution histories from hourly candlesticks using explicit UTC boundaries.
• Replaces stale settlement-date prices with the final binary outcome when a market is finalized.
• Falls back to Kalshi's historical endpoints for archived markets.
• Handles temporarily or permanently missing markets without failing the entire batch.
• Refreshes market status and timing immediately before curation so inactive markets are excluded.

Review feedback addressed

The latest implementation incorporates the relevant review feedback:

• I1: Only unresolved questions have their existing resolution histories downloaded during updates.
• T1: The question-cap test uses a small monkeypatched limit rather than exercising the production-sized cap.
• K1: Repeated sibling titles are disambiguated while authoritative resolution rules remain the contract.
• K2: Structured earliest/latest resolution bounds are normalized and refreshed before curation.
• K3: Hourly candles and explicit UTC as-of selection prevent look-ahead at daily boundaries.
• K4: Finalized questions receive an exact terminal 0 or 1 settlement value.
• K5: Archived-market routing, historical price normalization, bounded missing-market retries, and quarantine behavior are implemented.
• K6: Only active child markets enter discovery, and curation refresh removes markets that are no longer active.

The implementation deliberately does not attempt to infer contract semantics from tickers or parse arbitrary free-text early-close conditions. Kalshi's primary and secondary rules remain the authoritative resolution criteria.

Validation

• Kalshi and schema-focused tests: 110 passed
• Black, isort, flake8, pydocstyle, and git diff --check: passed
• No Kalshi failures in the full local test suite
• Personal GCP sandbox:
  • 3,219 unique fetch records
  • 3,219 matching question records
  • 3,219 structurally valid resolution-history files
  • No duplicate IDs, schema mismatches, invalid probabilities, discontinuous daily histories, or cutoff mismatches
  • Newly finalized live markets produced the expected terminal outcomes
  • Latest fetch and update Cloud Run jobs completed successfully

First-deployment note

Before the first update run, an empty kalshi_questions.jsonl must be manually seeded in the target question-bank bucket.

Deploy and run kalshi-fetch before kalshi-update-questions, because the update pipeline consumes the event_ticker and series_ticker fields emitted by the fetch job.

[nikbpetrov]

A few AI-generated comments. First set is ones I've verified myself that should be considered:

---

I1 — Update downloads the complete resolution backlog

Priority: P2
Location: src/orchestration/func_kalshi_update/main.py:26

load_existing_resolution_files(SOURCE) is called without ids=, which its own docstring warns
will list and download every source resolution file. The resulting frame dictionary is consulted
only in the unresolved-question loop. Resolved rows use a separate set of existing filenames.

Every resolved file is therefore downloaded and parsed nightly but never read. The cost grows
without bound as the source accumulates resolved questions and can eventually dominate the 3-hour,
4-GiB job.

Recommended fix: after loading dfq, pass only unresolved IDs:

ids = dfq.loc[~dfq["resolved"], "id"]
existing_resolution_files = _source_io.load_existing_resolution_files(SOURCE, ids=ids)

This bounds content downloads to the live pool while retaining the cheap all-ID existence listing
for regeneration checks.

T1 — The cap test uses the production limit and dominates the unit suite

Priority: P2
Location: src/tests/test_kalshi.py:751-762

test_caps_new_questions constructs _QUESTION_LIMIT - 1, or 4,999, unresolved rows. update()
then validates and iterates all of them, invoking two mocks per row. Isolated timing was about
35 seconds inside pytest and roughly 39 seconds wall-clock. The entire focused 71-test run took
roughly the same amount of time.

The test is checking boundary behavior, not production-scale performance.

Recommended fix: monkeypatch sources.kalshi._QUESTION_LIMIT to a small value such as two, use
one existing unresolved row and several new rows, and assert the same cap/retention contract.

---

Second set of AI comments I have not verified and looked into as they concern API specifics:

---

K1 — Stored questions omit or contradict the actual binary contract

Priority: P1
Location: src/sources/kalshi.py:172-174

update() stores only:

dfq.at[index, "question"] = market["title"]

Kalshi frequently defines an event-like title shared by many child contracts and puts the actual
Yes outcome in yes_sub_title and rules_primary. A full live pagination audit using the proposed
type, liquidity, open-interest, and date filters found 132 duplicate-title groups covering 789
qualifying markets.

Examples:

• Eight SpaceX thresholds all had the title “How many launches will SpaceX have in 2026?”, while
  their yes_sub_title values represented different thresholds.
• Thirty baseball-team contracts shared “Pro Baseball Playoff Qualifiers”, while the team was only
  identified by the child outcome.

There are also qualifying contracts whose title is factually inconsistent with their own rules.
On 2026-07-24, KXLEAVEPOWELLGOV-27JUN01 cleared both liquidity thresholds, but:

• title asked whether Jerome Powell would leave before August 1, 2026;
• yes_sub_title, rules_primary, ticker, and close_time specified June 1, 2027.

KXLEAVEPOWELLGOV-28JAN31 showed the same stale August 2026 title while its child contract and
rules specified January 2028.

The rules are later appended to an LLM forecaster's background, which partly mitigates omitted
outcome text for that one consumer. It does not fix the contract:

• metadata validation passes only row["question"], not the rules;
• public question identity and human-facing text remain ambiguous;
• duplicate titles represent different targets;
• stale titles and rules can directly contradict one another.

Recommended fix: build a self-contained binary question from market-specific data, including
the child yes_sub_title, and validate it against rules_primary, timing, and ticker semantics.
When Kalshi fields disagree, reject the market rather than guessing. Add tests with two child
markets sharing a title but having different Yes outcomes, plus a contradictory-title fixture.

K2 — Latest close time is not a safe earliest-resolution filter

Priority: P1
Locations:

• discovery: src/sources/kalshi.py:375-382
• stored curation date: src/sources/kalshi.py:178-180
• downstream filter: src/curate_questions/create_question_set/main.py:1137-1140

The code describes close_time as the resolution date and requires it to be at least ten days out.
For Kalshi, close_time can be a hard stop or postponement window substantially later than the
expected event and likely early close. The official lifecycle documentation distinguishes these
concepts.

Two market-specific live records demonstrate the contamination path:

Ticker	Expected event/expiration	Latest/close time	Contract behavior
KXLIGAMXGAME-26JUL24TIJLEO-TIJ	2026-07-25 05:00Z	2026-08-08 02:00Z	Closes after the game winner is declared; two-week close is a postponement/cancellation bound
KXBOXING-26JUL25JOSHUAPRENGA-JOSHUA	2026-07-26 00:00Z	2026-08-08 21:00Z	Closes after the match winner; later close permits rescheduling

July 23, 2026 was an actual ForecastBench curation day, with forecasts due around August 2. Both
markets passed the proposed close_time test even though their outcomes were expected to be known
more than a week before forecasts were produced. They could therefore consume frozen benchmark
slots with already-known outcomes. The base resolver may eventually nullify a question whose
resolution preceded its due date, but that happens after selection and does not prevent
contamination or restore the lost slot.

Blindly replacing close_time with expected_expiration_time is not safe either: the Powell ladder
records in K1 demonstrate that expected-expiration data can be stale across child contracts.

Recommended fix: model both bounds deliberately:

• treat a credible occurrence_datetime/expected_expiration_time or early-close condition as the
  earliest plausible resolution;
• retain close_time/latest expiration as the latest bound;
• reject markets whose credible earliest resolution is inside the freeze window;
• reject internally inconsistent title/rules/time records rather than selecting one stale field;
• persist a curation field representing the earliest plausible resolution, because the shared
  close-only curation filter cannot otherwise enforce the benchmark window.

Tests should cover a sports market with expected resolution before the minimum but a later
postponement close, an ordinary aligned contract, and a stale ladder record with a deliberate
conservative policy.

K3 — ET daily candles violate the UTC forecasting cutoff

Priority: P1
Locations:

• query cutoff: src/sources/kalshi.py:433-435
• candle labeling/fill: src/sources/kalshi.py:473-515
• downstream date joins: src/sources/_market.py:67-89

Kalshi's 1,440-minute candles end at midnight US Eastern time: 04:00 UTC during daylight time and
05:00 UTC during standard time. The implementation subtracts a fixed day and assigns the candle
ending at D+1 04:00/05:00Z to calendar date D.

ForecastBench's nightly job and forecast dates use UTC. This creates two failures from the same
misalignment:

1. The value is stale at the actual nightly run. At 00:00 UTC, the ET daily candle ending four
   or five hours later does not yet exist. The latest returned daily candle is shifted to the
   day-before-yesterday, and ffill() copies it into yesterday.
2. A later rebuild introduces look-ahead. After 04:00/05:00 UTC, that next ET candle exists.
   Backdating it to the previous UTC date assigns the first four or five hours of the next UTC day
   to the benchmark deadline. The same historical date can therefore change retroactively and
   include trades placed after the deadline.

A live comparison for KXLIGAMXGAME-26JUL24TIJLEO-TIJ made the error measurable:

• an hourly query ending exactly 2026-07-24 00:00Z gave the true July 23 UTC-end value, 0.5000;
• the daily query available at that same cutoff ended 2026-07-23 04:00Z at 0.5100;
• the current shift labels that older 0.5100 candle as July 22, then forward-fills 0.5100 into
  July 23 instead of using 0.5000;
• after the next ET boundary, the new daily candle can be mapped backward into July 23 even though
  it contains post-midnight UTC trading.

This directly affects market_value_on_due_date and
market_value_on_due_date_minus_one, used for imputation and naive forecasts. It also makes
missing-file reconstruction dependent on what time of day it is run.

Recommended fix: use hourly or finer candles and select the last observation at or before each
UTC day boundary, with an explicit as-of rule. Merely moving the job after 05:00 UTC fixes the stale
nightly read but not post-deadline look-ahead, so it is not sufficient by itself. Add a regression
proving that no observation after D 23:59:59Z can be stored as D.

This replaces the old report's L7. The old appendix claim that subtracting 24 hours was “safe”
proved only that it recovered an ET calendar label across DST; it did not establish UTC
no-look-ahead and was incorrect for ForecastBench.

K4 — Finalization can permanently preserve a nonterminal probability

Priority: P1
Location: src/sources/kalshi.py:462-466

The early return checks only whether the existing file reaches the resolution date:

if last_date >= cutoff:
    return existing_df

That is insufficient when a market has just transitioned from active/determined to finalized. An
unresolved run may already have written or forward-filled a market probability on the eventual
settlement date. On the first finalized run, last_date >= resolved_date returns that same file
before appending the terminal 0/1 result.

This was reproduced with a finalized Yes market settling on January 13 and an existing January 13
row of 0.63. _build_resolution_df returned the same object, never fetched candles, and retained
0.63. update() marked the question resolved and skipped the file upload because the object was
unchanged. Later runs see an existing resolution file and never enter missing-file regeneration.

MarketSource._resolve reads the last value as the final outcome. Since 0.63 is neither zero nor
one, it logs a warning and sets resolved_to to NaN, permanently discarding an otherwise valid
resolved question.

Recommended fix: the resolved early return must verify the terminal row represents the current
market result, not merely that a date exists. On transition to finalized, always insert/replace the
terminal resolution row (or rebuild deterministically). Add a regression with a same-date stale
price and both Yes and No outcomes.

K5 — Historical routing and 404 lifecycle are conflated

Priority: P2
Locations:

• unresolved handling: src/sources/kalshi.py:164-169, :229-242
• missing-file regeneration: src/sources/kalshi.py:208-227
• live detail/candles only: src/sources/kalshi.py:398-442

Kalshi moves older settled market data behind historical endpoints. On 2026-07-24, the returned
historical cutoff was 2026-05-25. A concrete archived market,
KXHIGHNY-25DEC08-T40, returned:

• 404 from the live market-detail endpoint;
• 404 from the live candlestick endpoint;
• 200 from the historical market endpoint; and
• 200 from the historical candlestick endpoint.

Historical candles also use a different price shape (price.close rather than live
price.close_dollars), so fallback requires normalization.

This breaks the source's explicit missing-resolution-file regeneration path for older resolved
questions. It also means a question that was not updated before crossing the archive cutoff cannot
recover. Depending on which live record remains available during the moving boundary, a candle 404
can abort the update while a detail 404 is mislabeled as permanent delisting and swallowed.

The same MarketNotFoundError policy conflates two more cases:

• Kalshi documents that newly created markets can briefly return 404 because detail propagation is
  asynchronous; the code deliberately makes 404 nonretryable and drops the new row until a later
  nightly fetch.
• A genuinely missing existing unresolved row is kept forever with stale metadata. It remains
  resolved=False, is retried every night, consumes _QUESTION_LIMIT, and remains eligible for
  curation.

Recommended fix: distinguish transient propagation, historical partitioning, and genuine
permanent removal. Route/fallback to the historical detail and candle endpoints when appropriate,
normalize both candle schemas, retry documented transient 404s, and explicitly invalidate or remove
a confirmed permanent orphan rather than keeping it indefinitely.

K6 — Open events can include non-active child markets

Priority: P2
Locations: src/sources/kalshi.py:271-300, :350-383

The events request uses status=open, but Kalshi defines an event as open when at least one child
market is active. Official lifecycle examples and a live pagination snapshot both show mixed child
statuses under open events.

_market_qualifies() checks type, liquidity, open interest, and time, but not the child's own
status. A closed, determined, disputed, inactive, or finalized child can therefore enter the
candidate set if it meets the other conditions. Non-finalized but nontradeable children are
especially problematic because _is_resolved() recognizes only finalized; they can remain
eligible for a new question set even though no new forecast can trade against the market.

No non-active child in the reviewed live snapshot happened to pass every other filter, so this is a
contract-level bug rather than evidence of current stored corruption.

Recommended fix: require market["status"] == "active" during discovery. Also define how an
existing question that closes/determines before curation should be excluded while it awaits final
settlement.

[pythoryn]

Thanks for the detailed review. I have reproduced the core issues and I'm working through them. I'll keep the PR as a single amended commit and follow up with the final verification results.

[pythoryn]

I have now pushed the latest rebased implementation and updated the PR description.

I reviewed all of the feedback again and applied the relevant changes, including unresolved-only resolution downloads, the improved cap test, conditional child-market labels, structured timing normalization, UTC-safe price histories, exact terminal settlement values, archived-market fallback, and active-only discovery/curation.

I deliberately kept two recommendations bounded: the code does not attempt a universal semantic comparison between Kalshi titles and rules, and it does not parse arbitrary free-text early-close conditions. Kalshi's full primary and secondary rules remain available as the authoritative contract, while the implemented structured checks cover the reliable provider fields without adding a fragile parsing subsystem.

Validation completed successfully: 110 focused tests and all changed-file style checks passed, with no Kalshi failures in the full local suite. The latest personal-sandbox Cloud Run jobs also succeeded, and all 3,219 generated questions and resolution histories passed the data-quality checks.

This should now be ready for re-review.