A non-2xx response with a JSON body is recorded in the hashlog and skipped on later runs, so failed records are never re-sent

[mark124]

Thanks for lightbeam — the hashlog idempotency model and the structured run metadata are really useful. This is a delivery-gap report with a runnable repro and a suggested fix.

Summary

When state tracking is on (state_dir set), do_post records a payload's hash in the hashlog for a non-2xx response that returns a parseable JSON body — e.g. a 429, or a 5xx that persists across all retries. On a later run, do_send skips any hash already in the hashlog unless meets_process_criteria() says otherwise, and under a plain lightbeam send (no --force, --resend-status-codes, --older-than, or --newer-than) meets_process_criteria() returns False for a stored failure exactly as it does for a stored success. So a record that failed once is indistinguishable from a delivered one and is skipped on every subsequent normal run — it is never re-sent, and no signal separates it from a genuine delivery.

Impact

Records are silently dropped from the ODS on normal re-runs (undelivered, not destroyed — the source .jsonl is untouched). This is most likely under the documented Airflow usage, where re-running a failed task is the normal recovery path — and that plain re-run is exactly the one that skips the failed records. The stored status does enable recovery via --resend-status-codes, but only for an operator who already knows to reach for it.

Version

lightbeam 0.1.12 (repro also present on main @ 1e357bd). Requires state_dir configured (the documented setup that enables the hashlog).

Why 429 is the clean case

The RetryClient uses retry_all_server_errors=True, so 5xx responses are retried in-run (up to num_retries) and a transient 5xx never reaches do_post's handler. A 429, being a 4xx, is not covered by retry_all_server_errors — and (see the related bug below) the configured retry_statuses are dropped — so a 429 passes straight through to the handler, gets its JSON body recorded in the hashlog, and is skipped forever. A 5xx reaches the same fate only if it persists across every retry.

Root cause

In lightbeam/send.py :: Sender.do_post, the whole non-401 branch is guarded by if status != 401: (line 153), so a 401 never reaches the write (it loops back to refresh the token). Inside that branch:

if response.status not in [200, 201]:
    response_body = json.loads(body)          # a non-JSON body raises JSONDecodeError (a ValueError) -> caught at the
    ...                                        #   `except ValueError` below, so it never reaches the hashlog write
    if response.status == 400:
        raise ValueError("; ".join(messages))  # 400 short-circuits before the write, too
    else:
        self.lightbeam.num_errors += 1          # 429 / persistent-5xx / 403 / 409 fall through...

# update hashlog
if self.lightbeam.track_state:
    self.hashlog_data[data_hash] = (round(time.time()), response.status)   # ...to HERE — written for a non-2xx

So the hashlog write is reached for a terminal non-2xx except a 400, except a 401 (handled separately), and except any response whose body isn't valid JSON (that raises JSONDecodeError, a ValueError, caught below). Then on the next run, do_send (lines ~80-103) skips it, because Lightbeam.meets_process_criteria (lightbeam.py ~237) is:

return ( self.force
         or (self.older_than and tuple[0] < self.older_than)
         or (self.newer_than and tuple[0] > self.newer_than)
         or (len(self.resend_status_codes) > 0 and tuple[1] in self.resend_status_codes) )

With default flags every clause is falsy, so a stored (timestamp, 429) returns False — treated as done. After run 1 the failed record is indistinguishable from a delivered one and is counted in the same num_skipped bucket, so nothing downstream separates the two.

Minimal reproduction (no live API needed)

import types
from lightbeam.lightbeam import Lightbeam
from lightbeam import hashlog

# a plain `lightbeam send` — no --force / --resend-status-codes / --older-than / --newer-than
flags = types.SimpleNamespace(force=False, older_than="", newer_than="", resend_status_codes="")

payload = '{"studentUniqueId": "12345"}'
h = hashlog.get_hash(payload)

# state after run 1: this record got a 429, which do_post wrote to the hashlog
hashlog_after_run_1 = {h: (1_700_000_000, 429)}

print("seen before? ", h in hashlog_after_run_1)                               # True
print("will resend? ", Lightbeam.meets_process_criteria(flags, hashlog_after_run_1[h]))  # False
# => the 429'd record is counted as skipped and never re-sent.

Expected: a record whose last status was a failure is retried on the next run (or at least not recorded as "handled").
Actual: it is stored like a success and skipped on every subsequent plain run.

Suggested fix

Distinguish transient from permanent. A transient failure (429, 5xx) should be resendable by default — ideally retried in-run once the retry bug below is fixed, or at minimum treated as "not done" by meets_process_criteria. A permanent failure (403/409/422) can be recorded but should be surfaced, not silently skipped. The invariant worth enforcing: a record is "done" only once the API has confirmed it.

Related bug (compounding)

api.py:94 builds the retry client as:

statuses=self.lightbeam.config['connection']["retry_statuses"].append(401)

list.append(...) returns None, so statuses=None reaches ExponentialRetry and the configured retry_statuses (429/500/501/503/504) are dropped — a 429 isn't retried at all. It also mutates config['connection']['retry_statuses'] in place (appending 401 on every call). Fixing this so 429/5xx are actually retried removes most of the transient failures that trigger the skip above.

Note

I hit both of these while auditing lightbeam for silent-failure modes; I have runnable tests pinning each and would happily open a PR with fixes + regression tests if useful.

[mark124]

On the retry_statuses side note in here, I went ahead and opened #93 for that one since it was pretty clear cut.

The main thing in this issue is more of a design call though, so I didn't want to just push a change without asking. Should a transient failure (a 429, or a 5xx that outlives the retries) get left out of the hashlog entirely, or written but treated as not-done by meets_process_criteria so the next run picks it back up? Both work, it just depends how you think about what the hashlog is tracking.

Let me know which way you'd want it and I'll put up a PR. And thanks for lightbeam, the hashlog idea itself is genuinely nice, this is really just about what counts as done.