Benchmark Helico on real MarinFold contact predictions (exp199 model)

[timodonnell]

Why

PR #10 established that a
genuinely MSA-free Helico, given contacts at 60% precision / 60% recall,
matches Protenix-with-MSAs on FoldBench (0.824 vs 0.837, Δ = −0.013 ± 0.026,
n.s., n=27 paired protein targets). Degrading a perfect map to 60/60 costs only
0.012 lDDT.

But those false positives are synthetic. They are drawn uniformly from the
eligible region by helico.contacts.sample_conditioning. Real predictor errors
are spatially correlated and cluster near true contacts — geometrically
plausible near-misses that the model cannot reject as inconsistent with the rest
of the map. Uniform-random false contacts are the easy case.

So the current result shows that a predictor at 60/60 suffices. This issue
tests whether MarinFold at its actual operating point suffices, which is the
question that decides the project.

Inputs

• Folding model: contacts-msafree-01 step 6000 —
  timodonnell/helico on the Hub,
  or /ckpts/contacts-msafree-01/final.pt on the helico-checkpoints Modal
  volume. MSA-free at training and inference.
• Contact predictor: MarinFold's current best, the 1.5B contacts-v1 model
  from exp199 (Eric's).
• Targets: the same FoldBench protein targets already used in PR Contact-conditioned folding: fold from pyconfind contacts instead of MSAs #10, so
  every number is paired against the existing synthetic-noise arms.

Modes to evaluate

1. Top-n contacts, pairwise vs rollout

MarinFold has two established inference recipes, and they are not
interchangeable — from exp180/build_dataset.py, the same checkpoint scores
~0.086 higher under rollout than under pairwise:

• pairwise (exp89): autoregressive P(contact), symmetrised.
• rollout (exp82, settled): n=100 rollouts + document resampling +
  pairwise tie-break, top-k off (#142).

Take the top-n ranked contacts from each, for n ∈ {L/5, L/2, L} where L is
the number of protein residues. That is 2 recipes × 3 values of n = 6 arms.

Feed them as a truncated top-k list: listed pairs → CONTACT_PRESENT, everything
else → CONTACT_UNKNOWN. Do not mark unlisted pairs CONTACT_ABSENT — the
model was trained on truncated lists precisely because an unlisted pair means
"did not make the cut", not "not a contact".

2. Actual rollouts, distribution of accuracies

Rather than aggregating the 100 rollouts into one ranking, feed individual
rollouts through Helico and report the distribution of resulting lDDT — not
just the mean. Worth extracting:

• spread across rollouts for a fixed target (how much does contact sampling
  matter to the final structure?)
• whether Helico's own confidence (ranking_score / pLDDT) correlates with
  structural accuracy across rollouts. If it does, rollout selection without
  ground truth becomes possible, which is a materially different capability
  from anything measured so far.
• best-of-N vs mean-of-N, as a headroom diagnostic. Note MarinFold already
  tracks oracle_best_of_100 as an explicitly non-deployable upper bound;
  Helico-side selection would be deployable if the correlation holds.

Suggest ~20 rollouts × ~27 targets to start, and scale if the spread is
interesting.

The comparison that actually answers the question

For every real-contact arm, measure MarinFold's realised precision and recall
against the pyconfind ground-truth map, then run a matched synthetic arm at
that same (p, r) via HELICO_BENCH_CONTACT_PRECISION/RECALL.

	real MarinFold contacts	synthetic at matched (p, r)
lDDT	?	?

• Gap ≈ 0 → error rate is what matters, our noise model is adequate, and
  the PR Contact-conditioned folding: fold from pyconfind contacts instead of MSAs #10 result transfers.
• Real is worse → error structure matters. That gap is the thing to fix,
  and the fix is in the noise model (sample false positives from near-miss
  pairs rather than uniformly), followed by a retrain.

This is the centrepiece. Without the matched-synthetic control, a low real-contact
score is uninterpretable — it could be error rate or error structure, and those
have completely different remedies.

Implementation sketch

1. Contact export from MarinFold. Ranked contact list per FoldBench target,
   for both recipes, plus the raw rollouts. exp20 and exp180 already score
   MarinFold on FoldBench, so target handling should mostly exist.
2. Index alignment. MarinFold emits sequence-position pairs; Helico needs a
   token-indexed (N_tok, N_tok) matrix. Both use pyconfind contacts-v1 so the
   contact definition matches, but the mapping still has to be verified —
   helico.bench.oracle_contact_state does the analogous re-indexing from a
   ground-truth structure and is the model to follow. Add a test asserting a
   round-trip of the ground-truth map through the MarinFold export format
   reproduces oracle_contact_state exactly, otherwise an indexing bug will
   present as "real contacts don't work".
3. Bench loader. A --contacts-from <file> path in modal/bench.py that
   reads an external contact list instead of deriving one. Mostly parallel to
   the existing HELICO_BENCH_ORACLE_CONTACTS plumbing.
4. Scoring. Reuse the existing paired analysis; all arms share targets with
   the PR Contact-conditioned folding: fold from pyconfind contacts instead of MSAs #10 runs.

Watch out for

• MarinFold training contamination. FoldBench targets were checked against
  Helico's training set (0 overlap). MarinFold's training set is different and
  must be checked separately — otherwise its contacts on these targets are
  optimistic. Note also that Helico's own validation set turned out 38%
  verbatim-contaminated, so this failure mode is live in this codebase.
• Recipe labelling. Never mix pairwise and rollout numbers in one series;
  the ~0.086 offset would masquerade as a real effect.
• n=27 targets. Small. Report paired differences with standard errors, not
  bare means, and don't read single-target results.
• L definition. Fix whether L counts protein residues only, and whether
  interface contacts across chains count toward n, before generating anything.

Done when

• Real-contact lDDT for all 6 top-n arms, each with a matched-synthetic control.
• The rollout accuracy distribution, with the confidence-correlation check.
• A statement of whether error structure matters, and if so, by how much.
• Writeup appended to RESULTS_contact_conditioning.md and the design doc.

[timodonnell]

Phase 0 done: MarinFold's real operating point, measured

Not a new rollout run — MarinFold exp211
is already generating exactly the artifacts this issue needs, from the exp199
model, and was 552/554 complete. Reused rather than duplicated.

Its foldbench100 set is FoldBench's first 100 monomer_protein targets
(5sbj_A ↔ 5sbj-assembly1). Its per-target parquet gives (rollout, order, i, j)
— per-rollout contact sets with emission order, so both recipes fall out of the
same data.

Measured accuracy vs pyconfind ground truth (n=100)

recipe	top-n	precision	recall
rollout (vote-aggregated)	L	0.505	0.564
rollout	L/2	0.676	0.379
rollout	L/5	0.795	0.179
single rollout	L	0.406	0.394
single rollout	L/2	0.406	0.228
single rollout	L/5	0.383	0.087

Ground truth is exp211's gt_universe.jsonl — pyconfind [i, j, degree] triples
in the same 0-indexed residue space as the predictions — with Helico's own
filters applied on top (degree >= MIN_CONTACT_DEGREE, |i-j| >= MIN_SEQ_SEPARATION)
so it matches what the folding model was trained against. Script:
.agents/project/data/measure_marinfold_accuracy.py.

Two things this changes

The 60/60 synthetic arm was optimistic. At its best-recall setting the real
predictor is ~50% precision / 56% recall, not 60/60. The matched-synthetic
controls must use the measured points above, and the headline
"60/60 matches Protenix+MSA" result sits slightly above where MarinFold actually
operates today.

Vote aggregation matters a lot. The rollout recipe beats a single rollout by
~0.10 precision at top-L, and single-rollout precision is essentially flat (~0.40)
regardless of n — i.e. one rollout's emission order carries little ranking signal,
while cross-rollout vote counts carry a lot. Worth keeping the two arms strictly
separate, per the ~0.086 offset already documented in exp180.

Next

Existing Helico numbers cover only 7 of these 100 targets (they were the
monomer_protein slice of a 60-target run). FoldBench has 334 monomer targets, so
the bench can be pointed at the same 100 with --categories monomer_protein,
giving n≈100 paired rather than 27.

Remaining build: the --contacts-from loader, and the index-alignment round-trip
test asserting a ground-truth map exported through MarinFold's format reproduces
oracle_contact_state exactly.

[timodonnell]

Result: real predictor errors cost far more than the same error rate

98 paired FoldBench monomer targets, all MSA-free, all arms on an identical
target list. Every synthetic arm was generated at the precision/recall measured
for its real counterpart, so the real-vs-synthetic gap isolates error structure
from error rate.

arm	lDDT
no contacts	0.365
single rollout, top-L	0.551
real MarinFold, top-L/5	0.564
real MarinFold, top-L/2	0.610
real MarinFold, top-L	0.622
synthetic @ p=.795 r=.179	0.682
synthetic @ p=.676 r=.379	0.757
synthetic @ p=.505 r=.564	0.786
oracle contacts	0.824
Protenix + MSA	0.851

The answer

Error structure matters, a lot. At identical precision and recall, real
MarinFold errors cost:

budget	real − synthetic	t	worse on
top-L/5	−0.119 ± 0.018	−6.5	77/98
top-L/2	−0.147 ± 0.020	−7.5	79/98
top-L	−0.165 ± 0.020	−8.5	81/98

For scale: at top-L, degrading the oracle map to 50%/56% rate costs 0.037
lDDT, while swapping uniform errors for real ones costs a further 0.165 —
about 4.5× more. The noise model was the easy case, exactly as flagged when the
synthetic result was published.

What this means

Real MarinFold contacts do help substantially: +0.257 ± 0.020 over no
contacts (t=12.6, 87/98). But they land at 0.622 against Protenix+MSA's 0.851 —
a gap of −0.230 ± 0.019 (t=11.9). MSA-free folding driven by today's MarinFold
does not yet match MSAs, and the earlier "60/60 synthetic matches Protenix+MSA"
result does not transfer.

Two secondary findings:

• Vote aggregation is worth +0.071 ± 0.008 (t=8.5) over a single rollout at
  the same budget, consistent with its ~0.10 precision advantage.
• On this monomer set, Protenix+MSA edges out even oracle contacts by
  +0.028 ± 0.014 (t=1.9, marginal) — unlike the assembly set where they matched.

Where the remaining gap can be attacked

The 0.230 shortfall decomposes as ~0.028 (oracle vs MSA) + 0.037 (error rate) +
0.165 (error structure). The dominant term is structure, which points at two
concrete levers:

1. Fix the training noise model to sample false positives from near-miss
   pairs rather than uniformly — the model has never been trained against the
   error distribution it actually faces. This is the cheapest thing to try.
2. Improve MarinFold's error profile, not just its precision/recall.

Reproduce: experiments/marinfold_contacts/export_contacts.py, then
bench_mf2_* arms; figure from .agents/project/figures/marinfold_real_contacts.py.