bazel l1: LRU observation flusher + LRU index entry cap

[shreyas-blacksmith]

Summary

L1-resident half of the Bazel LRU accounting move: the L1 observes AC accesses and persists them as advisory artifacts; the web-side retention sweep (unchanged) consumes them.

• cache/lruflush: buffers AC-access closures (cache.LRUObserver) per tenant storage prefix, dedupes by AC hash, and flushes one JSONL artifact per prefix every 5 minutes plus a synchronous drain at shutdown. One aggregation map, one serial flush goroutine, direct PUTs, no retries.
• cache/s3proxy.PutArtifact: writes one artifact under a fully-composed key, routed by the request-scoped backend/bucket selection exactly like cache entries (so the sweep, which follows the namespace shard pin, can find it), tagged lru=true for the bucket ILM backstop. The data-plane circuit breaker is consulted read-only: artifact traffic fails fast on a sick shard but can never trip or heal the breaker customer traffic depends on (both directions regression-tested).
• main.go: explicitly opt-in — the flusher runs only on trusted-mode nodes with an S3 proxy AND BAZEL_REMOTE_LRU_ARTIFACTS=1. Dark by default so a routine binary roll can never activate it fleet-wide; rollout enables per node (staging first) and widens with evidence.

Failure model

Everything here is advisory: under any pressure the correct response is to drop observations, count the drop, and move on — nothing can stall or fail a cache request.

• Request path: RecordACAccess admits via TryLock; lock contention is a metered drop (contention), never a queued cache request. Full buffer (250k object refs process-wide) drops rather than flushing early. Oversized closures (>50k leaves) drop whole, never truncate.
• Flush pass: bounded by a 2-minute wall clock (including the shutdown drain; worst case ~2 passes ≈ 4 min from SIGTERM if one is mid-flight), with per-(endpoint, bucket) failure suppression — a stalled backend costs one 30s PUT deadline per pass, never N of them. Abandoned/failed windows are counted per closure.
• Memory: structurally bounded to the active window + one detached window + one serialized artifact; buffers detach O(1) under the lock.

Metrics

bazel_remote_lru_artifact_flush_total{trigger,result}, bazel_remote_lru_flush_observations_dropped_total{reason}, bazel_remote_lru_flush_closures_lost_total{reason}, bazel_remote_lru_flush_buffered_objects, bazel_remote_lru_flush_detached_objects, bazel_remote_lru_flush_pass_duration_seconds, flush size/entry histograms.

Deliberately not in this PR (trimmed after review): gRPC transport tuning (stream quotas, recv caps, keepalives), GetTree guards, and tree-validation byte caps. The L1 keeps stock inbound/outbound transport behavior; defensive limits return only with evidence they're needed.

Test plan

• go test -race on all touched packages; full go test ./...; repeated flusher race runs
• Bazel 8.4.1: all changed targets pass
• CI: lint + new go test -race workflow green on this head
• Unit coverage: access-order round-trip, dedupe/union, drop reasons (missing prefix, oversized closure, buffer full, contention), failed-upload window drop, pass deadline, per-backend suppression, blocked-sink memory bound, breaker no-trip/no-heal, per-tenant bucket routing, key collision, concurrent conservation
• Staging E2E (pipeline): cold-cache smoke traffic → periodic flushes → schema-v1 artifacts in MinIO, zero drops, clean logs (verified on the trimmed predecessor head)
• Staging smoke/soak on this exact head: real AC traffic drove a periodic flush (165 closures, 54 KB, ~53 ms pass), schema-valid artifact in MinIO, zero drops (incl. contention), zero closures lost, clean logs on both nodes

---

Also bundled: LRU index entry cap (--max_entries)

The other half of the L1 memory story: the in-memory LRU index costs ~270 bytes per resident entry regardless of blob size (key string, entry struct, list node, map slot), and the byte budget alone does not bound it — zero-byte blobs charge nothing and tiny blobs charge at most one 4 KiB block, so a byte-full cache's index metadata is effectively unbounded on small-blob workloads.

• disk.WithMaxEntries(n) / --max_entries / BAZEL_REMOTE_MAX_ENTRIES / yaml max_entries: when positive, SizedLRU evicts least-recently-used entries past the count cap, exactly like the byte budget but counting entries. Applied during startup index load too, so a directory holding more files than the cap is trimmed oldest-first on boot.
• Default 0 = no bound; the binary roll changes nothing until the L1 role sets the flag (planned: --max_entries 200000000 ≈ 54 GiB index ceiling on 128 GiB nodes).
• Metric: bazel_remote_disk_cache_max_entries_evictions_total distinguishes count-cap evictions from byte-budget ones.
• Tests: tail eviction order, overwrite neutrality (replacing an existing key never evicts), cap enforcement during startup load, and a measured per-entry metadata cost test (logs the constant used for sizing).

Both features in this PR are inert by default (flusher: BAZEL_REMOTE_LRU_ARTIFACTS=1; cap: --max_entries > 0), so one binary release carries both and each is enabled/tuned/killed independently via config.

[shreyas-blacksmith]

Validation performed

Unit / race / build (head a4b35eb):

• go test -race on all touched packages (cache/lruflush, cache/s3proxy, cache/disk, server, main); full go test ./...; flusher suite repeated 3x under race.
• Bazel 8.4.1: all changed targets pass.
• CI on this exact head: go test -race and lint green (the race workflow was added in this PR).

Behavioral pins (all in-tree tests):

• Nonblocking admission: contention drops instead of blocking the hit path; concurrent conservation (flushed + counted drops = observations).
• Flush pass: whole-pass deadline abandons remaining prefixes after one stalled PUT; per-(endpoint,bucket) suppression skips a failed backend's remaining artifacts; blocked-sink memory bound.
• Breaker isolation both directions: 10 artifact failures leave the data-plane breaker closed; an artifact success cannot reset a 4-failure streak (5th still opens).

Staging E2E (2-node staging L1 fleet, real Bazel traffic via cold-cache smoke builds):

• Pipeline verified end-to-end on the immediate predecessor head (4fdb83c, identical pipeline shape): periodic flushes (1,144 closures / 361 KB across two artifacts), artifacts landed under <tenantPrefix>lru/ in MinIO and passed schema-v1 validation, zero drops, clean journals, healthy gRPC throughout.
• Three consecutive binary rolls exercised the shutdown drain cleanly ("Draining buffered LRU observation artifacts" on every stop).
• Both staging nodes now run this exact head (binary sha256 c841abd6…) with the feature enabled via the new opt-in env; a smoke/soak against this head is running to collect the canary-gate numbers (prefix cardinality, pass duration, drop/loss rates).

Rollout posture: binary is opt-in (BAZEL_REMOTE_LRU_ARTIFACTS=1); the Ansible role defaults dark with staging opting in (FA 3c9983b70, pushed).

[shreyas-blacksmith]

Exact-head staging gate complete (a4b35eb, binary c841abd6…): cold-cache AC traffic (googletest build) drove a periodic flush on node 2 — 165 closures / 54,367 bytes in a ~53 ms pass — artifact schema-valid in MinIO (window_end 22:05:44Z). Zero observation drops including the new contention reason, zero closures lost, no suppression or pass-deadline trips, clean journals on both nodes. All canary-gate signals (prefix cardinality, pass duration, drop/loss rates) nominal.