Tracking: reduce test-suite execution time
Running the full suite currently takes >4 min sequentially and ~48s with --parallel on a modern Mac. Profiling (macOS, default bash 3.2.57, Apple Silicon, commit 4d80e7c) shows most of that is framework overhead, not test logic:
Measured baseline
- 100 trivial tests (
assert_true true) take ~5.0s, i.e. ~48ms of pure framework overhead per test. Across ~1160 tests that is ~56s before any real test work.
- Per test the framework pays ~9 external-binary forks (~21ms) plus ~12-14
$(...) subshells (~5ms):
- 2x perl forks for clock reads (
src/clock.sh:88, called from src/runner.sh:1070,1097) because bash 3.2 has no EPOCHREALTIME and macOS date has no %N. ~2400 perl processes per suite run.
- 3x
base64 pipelines in bashunit::state::export_subshell_context (src/state.sh:239-246), even when title/hook message are empty.
- 1x
grep -B 2 | sed data-provider probe per test (src/helpers.sh:310-314), re-reading the test file from disk each time.
- 1x
base64 -d decode (src/runner.sh:1110) and 1x rm -rf glob cleanup (src/globals.sh:70) per test.
- Acceptance tests fork
./bashunit as a subprocess 258 times (52 of 59 acceptance files). Each is a full cold start whose inner tests pay the same per-test tax again. This dominates the remaining sequential minutes.
--parallel spawns with no default job cap (BASHUNIT_PARALLEL_JOBS=0; bashunit::runner::wait_for_job_slot short-circuits at src/runner.sh:293), so ~48s is essentially total CPU work / cores plus oversubscription overhead.
Child issues
Suggested order (independent, but small and mechanical first):
Expected combined effect on this machine: sequential >4min toward ~2-2.5min, parallel ~48s toward ~25-30s. Re-run the shared benchmark after each merge and update this issue.
Constraints for all work
- Bash 3.0+ compatibility is non-negotiable (no
declare -A, no [[ ]], no ${var,,}, no negative array indexing, no &>>). The perf target machine IS bash 3.2 on macOS.
- TDD, quality gate (
./bashunit tests/, ./bashunit --parallel tests/, make sa, make lint, shfmt -w .) on every PR.
- One issue per PR; no batching unrelated changes.
Shared benchmark protocol
Run before and after each change, report numbers in the PR:
mkdir -p /tmp/bashunit-bench
for i in $(seq 1 100); do echo "function test_noop_$i() { assert_true true; }"; done > /tmp/bashunit-bench/noop100_test.sh
time ./bashunit --no-parallel /tmp/bashunit-bench/noop100_test.sh # baseline: ~5.0s
# fork census (external binaries per run)
PS4='+X ${BASH_SOURCE##*/}:$LINENO ' bash -x ./bashunit --no-parallel /tmp/bashunit-bench/noop100_test.sh \
2>/tmp/bashunit-bench/trace.txt >/dev/null
grep -oE ' (perl|base64|grep|sed|mktemp|rm|date) ' /tmp/bashunit-bench/trace.txt | sort | uniq -c
Also considered and NOT filed for now: lazy-sourcing large optional modules (coverage.sh, learn.sh) at startup. Sourcing all of src/ measures only ~28ms, so the win across 258 nested cold starts is ~5-8s and it complicates the single-file release build. Revisit if the cheaper wins land and cold start is still the bottleneck.
Line references are valid at commit 4d80e7c and will drift.
Tracking: reduce test-suite execution time
Running the full suite currently takes >4 min sequentially and ~48s with
--parallelon a modern Mac. Profiling (macOS, default bash 3.2.57, Apple Silicon, commit 4d80e7c) shows most of that is framework overhead, not test logic:Measured baseline
assert_true true) take ~5.0s, i.e. ~48ms of pure framework overhead per test. Across ~1160 tests that is ~56s before any real test work.$(...)subshells (~5ms):src/clock.sh:88, called fromsrc/runner.sh:1070,1097) because bash 3.2 has noEPOCHREALTIMEand macOSdatehas no%N. ~2400 perl processes per suite run.base64pipelines inbashunit::state::export_subshell_context(src/state.sh:239-246), even when title/hook message are empty.grep -B 2 | seddata-provider probe per test (src/helpers.sh:310-314), re-reading the test file from disk each time.base64 -ddecode (src/runner.sh:1110) and 1xrm -rfglob cleanup (src/globals.sh:70) per test../bashunitas a subprocess 258 times (52 of 59 acceptance files). Each is a full cold start whose inner tests pay the same per-test tax again. This dominates the remaining sequential minutes.--parallelspawns with no default job cap (BASHUNIT_PARALLEL_JOBS=0;bashunit::runner::wait_for_job_slotshort-circuits atsrc/runner.sh:293), so ~48s is essentially total CPU work / cores plus oversubscription overhead.Child issues
Suggested order (independent, but small and mechanical first):
Expected combined effect on this machine: sequential >4min toward ~2-2.5min, parallel ~48s toward ~25-30s. Re-run the shared benchmark after each merge and update this issue.
Constraints for all work
declare -A, no[[ ]], no${var,,}, no negative array indexing, no&>>). The perf target machine IS bash 3.2 on macOS../bashunit tests/,./bashunit --parallel tests/,make sa,make lint,shfmt -w .) on every PR.Shared benchmark protocol
Run before and after each change, report numbers in the PR:
Also considered and NOT filed for now: lazy-sourcing large optional modules (
coverage.sh,learn.sh) at startup. Sourcing all ofsrc/measures only ~28ms, so the win across 258 nested cold starts is ~5-8s and it complicates the single-file release build. Revisit if the cheaper wins land and cold start is still the bottleneck.Line references are valid at commit 4d80e7c and will drift.