Add paired log audit and mechanism replays by waple0820 · Pull Request #4 · lexmount/lexmount-browser-lab

waple0820 · 2026-07-13T03:46:59Z

What changed

add a reproducible 64-task discordant log audit joining dataset, summaries, raw results, and Judge output
add a 12-task mechanism set with two counterbalanced replay rounds
add replay aggregation that excludes synthetic Judge failures instead of scoring them as zero
revise the main report to separate end-to-end backend performance from browser-engine causality

Finding

Of the 28-task net Lexmount advantage, 26 tasks align with an asymmetry in loser-side E1/E2/E3 site/access classifications (31 Lexmount-only versus 5 Local-only). In the raw logs, 25/31 Local E-class losers have captcha, HTTP denial, or net::ERR_* evidence. This supports an egress/IP/region/fingerprint availability mechanism, not a claim that remote Chromium is intrinsically better.

Targeted replays preserved strong Lexmount advantages on ASOS and 3DM (3/3 vs 0/3 each), but also preserved a Local advantage on 58.com (0/3 vs 3/3) and showed task-level flips elsewhere.

Validation

uv run pytest (15 passed)
uv run ruff check .
bash -n scripts/run_benchmark.sh
JSON/source consistency assertions
staged secret and machine-identifier scan

github-actions · 2026-07-13T03:47:24Z

Claude finished @waple0820's task in 1m 51s —— View job

PR Review

Gather context
Review scripts for bugs and security issues
Review tests
Post findings

Found three bugs in the new scripts. No security issues beyond a minor path note.

Bug 1 — KeyError crash when evaluations/results don't cover all paired tasks (`scripts/audit_paired_runs.py`, ~line 175)

task_ids is the intersection of the two summaries, but lexmount_evaluations[task_id] and lexmount_results[task_id] (and their local_ counterparts) are indexed directly with no existence check. If a task appears in both summaries but is absent from the evaluation or results files, this crashes with an unhandled KeyError.

# current — will KeyError if task_id missing from evals/results
lex_arm = arm_record(
    task_id,
    lexmount_summary,
    lexmount_evaluations[task_id],
    lexmount_results[task_id],
)

Fix: compute task_ids as the intersection of all four dicts, or use .get() with a sentinel and skip/log missing tasks. Fix this →

Bug 2 — `set.intersection` raises `TypeError` when both summary lists are empty (`scripts/summarize_replays.py`, ~line 78)

task_ids = sorted(
    set.intersection(*(set(summary["per_task"]) for summary in all_summaries)), key=int
)

set.intersection() called with no arguments (when all_summaries = []) raises TypeError: descriptor 'intersection' requires a 'set' object but received a 'generator'. This fires if both --lexmount-summary and --local-summary lists happen to be empty, or if summarize_replays is called from tests with empty inputs.

Fix:

sets = [set(s["per_task"]) for s in all_summaries]
task_ids = sorted(set.intersection(*sets) if sets else set(), key=int)

Fix this →

Bug 3 — NDJSON fallback in `load_json_records` re-raises on malformed lines (both scripts)

The outer try/except json.JSONDecodeError only catches the full-file parse. The fallback list comprehension has no per-line guard:

except json.JSONDecodeError:
    value = [json.loads(line) for line in text.splitlines() if line.strip()]
    #         ^^^^ unhandled JSONDecodeError if any line is malformed

A file that is neither valid JSON nor valid NDJSON (e.g. truncated on the last line) will surface as an unhelpful traceback rather than a clear error. Fix this →

Minor — `synthetic_evaluation_ids` resolves `run_dir` from summary JSON without sanitization (`scripts/summarize_replays.py`, ~line 28)

Path(summary["run_dir"]) is used to glob for eval files. If a summary JSON ever originates from an untrusted source, this is an arbitrary filesystem read. Not exploitable in the current pipeline (summaries are generated by the harness), but worth noting for future use.

Tests look correct for what they cover. The _summary fixture in test_summarize_replays.py correctly exercises the happy path, and the test_raw_log_indicators_ignore_status_digits_inside_urls case is a good regression guard for the HTTP status regex.

github-actions

Automated Claude review completed. Review run: https://github.com/lexmount/LexBrowserEnv/actions/runs/29222430108. Please check Claude's inline feedback and summary comment for details.

waple0820 · 2026-07-13T03:52:33Z

Addressed all three review findings:

paired audit now validates complete summary/evaluation/result coverage and raises a clear error listing missing task IDs
replay aggregation rejects empty arm inputs with an explicit contract
both NDJSON loaders now report the source path and malformed line number

Added regression tests for all cases. Re-generated audit and replay JSON are byte-identical to the committed outputs, so no experiment statistics changed. Full suite: 19 passed; Ruff and Bash checks pass.

github-actions · 2026-07-13T03:52:41Z

Claude finished @waple0820's task in 1m 46s —— View job

PR Review

Gather context
Review updated scripts for bugs and security issues
Verify previous bug fixes are correct
Post findings

All three previous bugs are correctly fixed. One new bug found.

Previous fixes — all correct

Bug 1 (KeyError): paired_task_ids() now validates all six dicts before any indexing — correct.
Bug 2 (empty set.intersection): The early raise ValueError when either arm list is empty ensures task_sets always has ≥ 2 elements before set.intersection(*task_sets) is called — correct.
Bug 3 (NDJSON per-line error): Both scripts now catch per-line JSONDecodeError and raise a ValueError with path and line number — correct.

Bug — `dataset[task_id]` not covered by the coverage check (`scripts/audit_paired_runs.py`, line 226)

paired_task_ids validates coverage across the six summary/evaluation/result dicts, but dataset is a seventh independently-loaded dict that is never cross-checked. For discordant tasks, dataset[task_id] is called with no existence guard:

# task_ids is the intersection of 6 dicts — but not dataset
metadata = dataset[task_id]   # KeyError if dataset is missing this task

If the dataset file is incomplete or filtered differently from the run files, this crashes with an unhelpful KeyError mid-loop. Fix this →

github-actions

Automated Claude review completed. Review run: https://github.com/lexmount/LexBrowserEnv/actions/runs/29222614237. Please check Claude's inline feedback and summary comment for details.

waple0820 · 2026-07-13T03:56:44Z

Added dataset coverage as the seventh validated source and a regression test for missing dataset rows. Re-generated paired audit remains byte-identical.

github-actions · 2026-07-13T03:57:18Z

Claude finished @waple0820's task in 1m 49s —— View job

PR Review

Gather context
Review updated scripts for bugs and security issues
Verify previous bug fix (dataset coverage) is correct
Post findings

Previous fix — correct

paired_task_ids() now includes dataset in the sources dict that drives the coverage check, so any task missing from the dataset file raises a clear ValueError before indexing. The regression test (test_paired_task_ids_reports_missing_dataset_row) correctly exercises this path.

Bug — `record["task_id"]` unguarded in `synthetic_evaluation_ids` (`scripts/summarize_replays.py`, line 40)

if benchmark.get("is_synthetic_failure"):
    synthetic.add(str(record["task_id"]))   # KeyError if key absent

load_json_records filters to dict records but does not guarantee any key is present. A synthetic-flagged eval record missing task_id raises an unhelpful KeyError here. The fix follows the same pattern used elsewhere — use .get():

if benchmark.get("is_synthetic_failure") and "task_id" in record:
    synthetic.add(str(record["task_id"]))

or raise a ValueError with the path and field name for clarity. Fix this →

No other bugs found. Tests cover the newly added validation paths correctly.

github-actions

Automated Claude review completed. Review run: https://github.com/lexmount/LexBrowserEnv/actions/runs/29222767340. Please check Claude's inline feedback and summary comment for details.

waple0820 · 2026-07-13T04:01:49Z

Guarded synthetic evaluation records missing task_id with a path-qualified ValueError and added the requested regression test. Replay output remains byte-identical.

github-actions · 2026-07-13T04:01:51Z

Claude finished @waple0820's task in 1m 30s —— View job

PR Review

Gather context
Review latest fix (task_id guard in synthetic_evaluation_ids)
Check for new bugs
Post findings

Previous fix — correct

synthetic_evaluation_ids now uses record.get("task_id") and raises a path-qualified ValueError when the field is absent or empty. The regression test correctly exercises this path.

Bug — unguarded key access in `load_dataset` and `load_evaluations` (`scripts/audit_paired_runs.py`, lines 54 and 58)

Both loader functions directly index a key from each record without a guard:

def load_dataset(path: Path) -> dict[str, dict[str, Any]]:
    return {str(record["id"]): record for record in load_json_records(path)}

def load_evaluations(path: Path) -> dict[str, dict[str, Any]]:
    return {str(record["task_id"]): record for record in load_json_records(path)}

load_json_records filters to dict objects but does not guarantee any specific key is present. A malformed dataset or evaluation file (e.g. a record missing "id" or "task_id") crashes with an unhelpful KeyError rather than a path-qualified error. This is the same class of bug as the one just fixed in synthetic_evaluation_ids.

Fix: raise a ValueError with the file path and record index when the required key is absent:

def load_dataset(path: Path) -> dict[str, dict[str, Any]]:
    out = {}
    for i, record in enumerate(load_json_records(path)):
        if "id" not in record:
            raise ValueError(f"{path}: record {i} is missing 'id'")
        out[str(record["id"])] = record
    return out

def load_evaluations(path: Path) -> dict[str, dict[str, Any]]:
    out = {}
    for i, record in enumerate(load_json_records(path)):
        if "task_id" not in record:
            raise ValueError(f"{path}: record {i} is missing 'task_id'")
        out[str(record["task_id"])] = record
    return out

Fix this →

No other bugs found.

github-actions

Automated Claude review completed. Review run: https://github.com/lexmount/LexBrowserEnv/actions/runs/29222939581. Please check Claude's inline feedback and summary comment for details.

github-actions · 2026-07-13T04:08:51Z

Claude finished @waple0820's task in 1m 40s —— View job

PR Review

Gather context
Verify previous bug fixes (load_dataset / load_evaluations guards)
Check for new bugs
Post findings

Previous fix — correct

load_dataset and load_evaluations now delegate to index_records, which uses .get(key) and raises a path-qualified ValueError with the record number when the key is absent. It also correctly detects duplicate key values. The regression tests cover both paths.

Bug — `load_results` uses bare `json.loads` with no path info on error (`scripts/audit_paired_runs.py`, line 76)

def load_results(run_dir: Path) -> dict[str, dict[str, Any]]:
    return {
        path.parent.name: json.loads(path.read_text(encoding="utf-8"))  # no error context
        for path in sorted((run_dir / "tasks").glob("*/result.json"))
    }

A malformed result.json raises a raw json.JSONDecodeError with no indication of which file caused it — the same class of unhelpful error that was fixed in load_json_records. Every other loader in these scripts now surfaces a path-qualified ValueError; load_results is the one exception. Fix this →

No other bugs found.

github-actions

Automated Claude review completed. Review run: https://github.com/lexmount/LexBrowserEnv/actions/runs/29223213513. Please check Claude's inline feedback and summary comment for details.

github-actions · 2026-07-13T04:14:58Z

Claude finished @waple0820's task in 3m 12s —— View job

PR Review

Gather context
Verify previous bug fix (load_results path-qualified error)
Check for new bugs
Post findings

Previous fix — correct

load_results now delegates to load_json_object(path) instead of bare json.loads, so a malformed result.json raises a ValueError including the file path. The regression test (test_load_results_reports_malformed_result_path) correctly exercises this path.

No new bugs found. All loaders in both scripts now produce path-qualified errors on bad input, and the coverage validation in paired_task_ids covers all seven data sources. The scripts are clean.

github-actions

Automated Claude review completed. Review run: https://github.com/lexmount/LexBrowserEnv/actions/runs/29223453377. Please check Claude's inline feedback and summary comment for details.

waple0820 · 2026-07-13T04:18:53Z

Final verification: 32 tests pass; Ruff and shell syntax checks pass; the full paired audit and targeted replay outputs regenerate byte-identically from the raw artifacts. Latest automated review found no new bugs.