internal/sqlsmith: TestSetups failed

[cockroach-teamcity]

internal/sqlsmith.TestSetups failed on master @ e2808826166146c4644b1c30b3b20a16d8d315d4:

=== RUN   TestSetups
    test_log_scope.go:172: test logs captured to: outputs.zip/logTestSetups2460148775
    test_log_scope.go:82: use -show-logs to present logs inline
=== RUN   TestSetups/rand-tables
    test_server_shim.go:168: automatically injected an external process virtual cluster under test; see comment at top of test_server_shim.go for details.

Parameters:

• attempt=1
• race=true
• run=2
• shard=4

Help

See also: How To Investigate a Go Test Failure (internal)

Improve this report!

Jira issue: CRDB-67115

[michae2]

/investigate

[github-actions]

Investigation: internal/sqlsmith.TestSetups

Investigated failure: issue body — EngFlow invocation c2975368…, run=2 shard=4
Failure SHA: e2808826166146c4644b1c30b3b20a16d8d315d4
Confidence: moderate–high that this is an out-of-memory / external process kill, not a code defect; high that it is not a bug in rand-tables.

What This Test Does

TestSetups iterates over every entry in sqlsmith.Setups; for each it starts a fresh test server, generates the setup's SQL, and executes each statement, failing if any statement errors. The rand-tables subtest generates random tables and types via randgen.RandCreateTables. The test shares its go_test target (shard 4) with the heavy, server-driven TestEPSCReproduces.

Where the Failure Occurs

The failure is a truncated subtest that never completed — not an assertion failure or SQL error. test.xml records TestSetups and TestSetups/rand-tables with "No pass/skip/fail event found for test", and test.log ends with the whole binary being killed:

=== RUN   TestSetups/rand-tables
    test_server_shim.go:168: automatically injected an external process virtual cluster under test; ...
I260824 12:24:08.556544 1 (gostd) testmain.go:140  [-] 1  Test //pkg/internal/sqlsmith:sqlsmith_test exited with error code -1

The rand-tables server was healthy and mid-bootstrap (last node log at 12:24:07, an in-progress CREATE DATABASE migration step) ~1 second before the kill. There is no panic, no data race report, no goroutine dump, and no OOM/timeout marker anywhere in the captured artifacts (test.log, test.xml, or the node logs in outputs.zip). That signature — abrupt external termination with exit code -1 and no Go-level stack — is a killed process, not a code failure in rand-tables.

Analysis

Hypothesis 1 — OOM kill under concurrent-run memory pressure (most likely).

Shard 4 ran 3× (--runs_per_test, run=1,2,3). The timing decisively rules out a timeout and points to a memory kill:

	EPSC total	TestSetups	Outcome
run 1	1115.31s	680.32s (all 6 subtests)	PASS
run 3	885.92s	613.51s (all 6 subtests)	PASS
run 2	786.30s	killed during rand-tables (~157s in)	KILLED (exit -1)

Run 2 was the fastest run of TestEPSCReproduces and was killed only ~16 minutes into the process (start 12:08:02, kill 12:24:08), whereas runs 1 and 3 ran ~25–30 minutes to completion and passed. A per-process timeout cannot explain a kill that fires earlier than two sibling runs that survived much longer — this rules out the timeout hypothesis that the prior investigation of #173052 leaned toward.

The most consistent explanation is an OOM kill: the three race-instrumented runs of this heavy shard execute concurrently on the same executor and compete for RAM; run 2 lost the memory race and was SIGKILLed by the kernel/cgroup, leaving no Go stack. This is directly corroborated by the near-identical prior failure #173052, where a SQL Queries collaborator confirmed via an internal dashboard that the process was "OOMed" and noted OOM frequency had been rising.

Hypothesis 2 — generic non-OOM SIGKILL / executor eviction (lower confidence). A non-OOM kill (executor preemption, cgroup CPU-time cap, etc.) would produce the same stackless exit--1 signature. Downweighted relative to OOM given the team's explicit OOM confirmation on the sibling failure, but not fully excludable from the captured artifacts alone (the EngFlow action's exit-reason metadata is not in the downloadable artifacts).

Ruled out — a defect in rand-tables or its generated SQL. The subtest was killed before executing any setup statement (server still bootstrapping), and rand-tables passed in runs 1 and 3 of the same shard (103s, 155s). No error, panic, or data race was produced.

Contributing factor (deterministic) — sharding co-locates the two heaviest test groups. rules_go sharding is deterministic, so shard 4 chronically carries both TestEPSCReproduces (unsplittable, server-heavy, ~13–18 min under race) and the full TestSetups suite. That maximizes the concurrent memory footprint on the executor whenever multiple runs of shard 4 overlap. See the sharding comment in BUILD.bazel:69-73. This config is unchanged since the #173052 investigation.

Recommendations

• Treat this as infrastructure resource exhaustion (OOM) rather than a code defect; a retry will very likely pass (2 of the 3 runs here passed). This should likely not remain a release-blocker on its own merits — it is a known, recurring env/OOM flake for this target, not a product bug.
• To reduce recurrence, reduce the peak concurrent memory of this target under race:
  • Split TestEPSCReproduces into its own go_test target so it no longer shares a process/shard budget with TestSetups, and/or increase shard_count so no single process carries both heavy groups.
  • Consider skip.UnderRace for the heaviest EPSC subtests, or lowering --runs_per_test/concurrency for this race target so three memory-heavy runs don't overlap on one executor.
• If the OOM trend is broader (the collaborator on internal/sqlsmith: TestSetups failed #173052 noted rising OOM frequency), the SQL Queries / Test Eng teams may want to look at executor memory limits for race-instrumented size=large targets.

Related Issues and PRs

• internal/sqlsmith: TestSetups failed #173052 (internal/sqlsmith: TestSetups failed, CLOSED) — same failure mode: identical shard-4 truncated-rand-tables kill (exit -1, no stack), and a collaborator explicitly confirmed it was OOMed. This is the strongest signal that internal/sqlsmith: TestSetups failed #173815 is a recurrence of the same OOM.
• Older TestSetups failures (internal/sqlsmith: TestSetups failed #146178, internal/sqlsmith: TestSetups failed #125166, internal/sqlsmith: TestSetups failed #122616, internal/sqlsmith: TestSetups failed #122520, internal/sqlsmith: TestSetups failed #122425, internal/sqlsmith: TestSetups failed #114962, internal/sqlsmith: TestSetups failed #112372, …) are all CLOSED and mostly represent different failure modes (e.g. internal/sqlsmith: TestSetups failed #146178 was a genuine nil pointer SIGSEGV with a full stack). None match this clean external-kill signature.
• The EPSC oracle tests + shard_count=8/size=large config were introduced on 2026-07-29 (042de065d03, a7fd0b71113), which grew shard 4's memory/time budget shortly before these failures began.

Timeline

• 2026-07-29: EPSC oracle tests + sharding/size=large added to the sqlsmith package.
• 2026-08-01: failure internal/sqlsmith: TestSetups failed #173052 (shard 4 run 3) — team-confirmed OOM.
• 2026-08-24 12:08:02: shard 4 run 2 begins.
• 2026-08-24 12:24:08: binary killed (exit -1) during TestSetups/rand-tables startup, ~16 min in; runs 1 and 3 ran ~25–30 min and passed.

Detailed evidence and investigation notes

Log Excerpts

test.xml (run 2), the two truncated cases:

<testcase name="TestSetups" time=""><error message="No pass/skip/fail event found for test">
=== RUN   TestSetups ...</error></testcase>
<testcase name="TestSetups/rand-tables" time=""><error message="No pass/skip/fail event found for test">
=== RUN   TestSetups/rand-tables
    test_server_shim.go:168: automatically injected an external process virtual cluster under test ...</error></testcase>

Last node-log lines (outputs.zip/logTestSetups2460148775/sqlsmithtest.log) show a healthy server executing CREATE DATABASE migration steps at 12:24:07, ~1s before the kill — nothing hung inside rand-tables.

Searches across all downloaded artifacts for DATA RACE, panic:, fatal error, out of memory, SIGKILL, signal killed, cannot allocate, test timed out, and goroutine N [ returned no matches — consistent with an external SIGKILL, not a Go-level failure.

Per-run shard-4 timing (from test.xml): run 1 EPSC 1115.31s + TestSetups 680.32s (pass); run 3 EPSC 885.92s + TestSetups 613.51s (pass); run 2 EPSC 786.30s, killed ~157s into TestSetups. The killed run was the shortest-lived, which is inconsistent with a per-process timeout and consistent with a memory-pressure kill among concurrent runs.

Code References

• setup_test.go:31 — TestSetups loop (fresh server per subtest).
• BUILD.bazel:54-94 — size = "large", shard_count = 8; comment explaining EPSC-driven sharding.

Things Ruled Out

• Bug in rand-tables / generated DDL: killed before executing any setup statement (server still bootstrapping); passed in runs 1 and 3 of the same shard.
• Data race: no WARNING: DATA RACE in any artifact despite race=true.
• Per-process timeout: the killed run was shorter-lived (~16 min) than the two sibling runs that passed (~25–30 min). Decisively excludes a timeout.
• Recurrence of prior TestSetups SIGSEGV (internal/sqlsmith: TestSetups failed #146178): that had a full Go stack; this has none.

Other Failure Occurrences

This issue thread has only the single (initial) failure occurrence (plus the /investigate trigger). Runs 1 and 3 of the same shard/invocation passed, underscoring the flaky/environmental (OOM) nature of run 2.

Tooling Feedback

• The EngFlow helper had to be invoked via .claude/skills/engflow-artifacts/run.sh; the direct python3 .../engflow_artifacts.py form shown in the prompt's Step 4 is not in the allowlist and was denied. Worth updating the prompt to the run.sh wrapper (this was also flagged in the internal/sqlsmith: TestSetups failed #173052 investigation).
• File-inspection tools (Read/Grep/find/cp) are restricted to the repo working directory, so artifacts downloaded to /tmp were unreadable. I had to re-download directly into artifacts/. The prompt's Step 4 examples use /tmp/engflow-artifacts as the outdir — recommend using a path under the working tree.
• The EngFlow action result metadata (exit reason: OOM vs TIMEOUT vs signal, and peak memory) is not in the downloadable artifacts. Access to it would have made the OOM diagnosis conclusive rather than moderate–high confidence; today it relies on the timing analysis plus the team's OOM confirmation on the sibling issue.

---

Workflow run

Was this investigation helpful? Leave a 👍 or 👎 on this comment.

---

Generated by claude-opus-4-8 at high effort. Use /investigate fable-5 max to go deeper.

Workflow run · Was this investigation helpful? Leave a 👍 or 👎 on this comment.