Skip to content

perf: reduce test-suite execution time (tracking) #761

Description

@Chemaclass

Tracking: reduce test-suite execution time

Running the full suite currently takes >4 min sequentially and ~48s with --parallel on a modern Mac. Profiling (macOS, default bash 3.2.57, Apple Silicon, commit 4d80e7c) shows most of that is framework overhead, not test logic:

Measured baseline

  • 100 trivial tests (assert_true true) take ~5.0s, i.e. ~48ms of pure framework overhead per test. Across ~1160 tests that is ~56s before any real test work.
  • Per test the framework pays ~9 external-binary forks (~21ms) plus ~12-14 $(...) subshells (~5ms):
    • 2x perl forks for clock reads (src/clock.sh:88, called from src/runner.sh:1070,1097) because bash 3.2 has no EPOCHREALTIME and macOS date has no %N. ~2400 perl processes per suite run.
    • 3x base64 pipelines in bashunit::state::export_subshell_context (src/state.sh:239-246), even when title/hook message are empty.
    • 1x grep -B 2 | sed data-provider probe per test (src/helpers.sh:310-314), re-reading the test file from disk each time.
    • 1x base64 -d decode (src/runner.sh:1110) and 1x rm -rf glob cleanup (src/globals.sh:70) per test.
  • Acceptance tests fork ./bashunit as a subprocess 258 times (52 of 59 acceptance files). Each is a full cold start whose inner tests pay the same per-test tax again. This dominates the remaining sequential minutes.
  • --parallel spawns with no default job cap (BASHUNIT_PARALLEL_JOBS=0; bashunit::runner::wait_for_job_slot short-circuits at src/runner.sh:293), so ~48s is essentially total CPU work / cores plus oversubscription overhead.

Child issues

Suggested order (independent, but small and mechanical first):

Expected combined effect on this machine: sequential >4min toward ~2-2.5min, parallel ~48s toward ~25-30s. Re-run the shared benchmark after each merge and update this issue.

Constraints for all work

  • Bash 3.0+ compatibility is non-negotiable (no declare -A, no [[ ]], no ${var,,}, no negative array indexing, no &>>). The perf target machine IS bash 3.2 on macOS.
  • TDD, quality gate (./bashunit tests/, ./bashunit --parallel tests/, make sa, make lint, shfmt -w .) on every PR.
  • One issue per PR; no batching unrelated changes.

Shared benchmark protocol

Run before and after each change, report numbers in the PR:

mkdir -p /tmp/bashunit-bench
for i in $(seq 1 100); do echo "function test_noop_$i() { assert_true true; }"; done > /tmp/bashunit-bench/noop100_test.sh
time ./bashunit --no-parallel /tmp/bashunit-bench/noop100_test.sh   # baseline: ~5.0s

# fork census (external binaries per run)
PS4='+X ${BASH_SOURCE##*/}:$LINENO ' bash -x ./bashunit --no-parallel /tmp/bashunit-bench/noop100_test.sh \
  2>/tmp/bashunit-bench/trace.txt >/dev/null
grep -oE ' (perl|base64|grep|sed|mktemp|rm|date) ' /tmp/bashunit-bench/trace.txt | sort | uniq -c

Also considered and NOT filed for now: lazy-sourcing large optional modules (coverage.sh, learn.sh) at startup. Sourcing all of src/ measures only ~28ms, so the win across 258 nested cold starts is ~5-8s and it complicates the single-file release build. Revisit if the cheaper wins land and cold start is still the bottleneck.

Line references are valid at commit 4d80e7c and will drift.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    Status
    No status

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions