Skip to content

Fault injection and chaos tests for the pool - #929

Draft
benoitc wants to merge 4 commits into
masterfrom
test/pool-fault-injection
Draft

Fault injection and chaos tests for the pool#929
benoitc wants to merge 4 commits into
masterfrom
test/pool-fault-injection

Conversation

@benoitc

@benoitc benoitc commented Aug 11, 2026

Copy link
Copy Markdown
Owner

Builds on #928 and carries its two commits, so this should land after it (a rebase drops them). Only the last commit is new here.

Integration tests only exercise servers that answer, so nothing covered a connection that stalls, crashes, or dies mid-checkout, which is the shape of #927. Adds:

  • hackney_fault_transport, a transport that behaves like hackney_tcp until a fault is armed per callback ({sleep, Ms}, {slow_error, Ms}, {hang, Ms}, {error, Reason}, crash), so faulted connections still talk to a real server.
  • hackney_crash_sentinel, a logger handler that makes a dead pool visible under error_logger:tty(false).
  • hackney_pool_fault_tests, one fault per scenario across the checkout entry points, asserting the caller gets an error, the pool is the same pid it was, nothing is left checked out or queued, and no connection leaked.
  • hackney_pool_chaos_tests, concurrent workers with a monkey rotating the fault matrix underneath, scalable for a release soak with HACKNEY_CHAOS_WORKERS and HACKNEY_CHAOS_ROUNDS.
  • hackney_pool_safety_tests, which walks the compiled abstract code and fails on any call from the pool into a connection process that is not inside a try.

The structural test found four live instances of the same class as #927: three unguarded hackney_conn:set_owner/2 calls and the prewarm dial. They now go through guarded helpers.

The hang scenario found a second problem that #928 makes more likely: stop_conn used gen_statem:stop/1, which waits forever, so after a dial timed out the pool blocked on stopping the wedged connection for as long as the transport took to return. Adds hackney_conn:stop/2 and bounds the pool's stop at 100ms, killing past the deadline. That scenario went from 1500ms to 152ms.

Full eunit is green (1090), xref and dialyzer clean. The fault suite fails on plain master in the expected ways.

aboroska and others added 4 commits August 11, 2026 09:35
Convert a connection call timeout into a checkout error. The pool
must not terminate just because a DNS/TCP/TLS attempt outlives its timeout.

Fixes: #927
Only exit:{timeout, _} was caught, so a transport raising or the conn
being killed while dialing still took the pool down. Catch any exit from
the connect call and return it as a checkout error. The caller already
stops the conn on every error return.
Integration tests only exercise servers that answer, so nothing covered a
connection that stalls, crashes, or dies mid-checkout, which is how #927
reached a release. Adds a transport that can be told to misbehave, a crash
sentinel that makes a dead pool visible under error_logger:tty(false), a
fault matrix over the checkout entry points, a randomized chaos run, and a
structural test that fails on any unguarded call from the pool into a
connection process.

The structural test found three unguarded set_owner calls and the prewarm
dial; they now go through guarded helpers. Stopping a connection from
inside the pool is bounded too: a conn wedged in a transport call used to
hold the pool for as long as the transport took to return.
Extends the fault harness to the multiplexed checkout paths, where a bad
connection is worse than on the HTTP/1 path: HTTP/2 and HTTP/3 connections
are shared, so every caller for that host goes through the same probe.

h2_conn_usable claimed a short timeout but called get_state/1, which waits
the default 5s, so one wedged connection stalled the pool for 5s per
checkout. The is_ready, checkin_info and set_owner probes had the same
5s exposure. All four now take an explicit timeout and the pool passes 250ms:
a connection that cannot answer a question about its own state at once is
unusable to the pool, and waiting on it blocks every caller behind it.

The new suite wedges a registered connection with sys:suspend/1 and fails
if the pool waits on it. Before the change that scenario hit the eunit
timeout; it now answers none in 252ms.
@benoitc

benoitc commented Aug 11, 2026

Copy link
Copy Markdown
Owner Author

Extended to the multiplexed checkout paths in 721cf78.

h2_conn_usable/1 carried a comment claiming a short timeout but called hackney_conn:get_state/1, which waits the 5s default, so a single wedged HTTP/2 connection stalled the pool for 5s on every checkout for that host. is_ready, checkin_info and set_owner had the same exposure. All four now take an explicit timeout and the pool passes 250ms.

hackney_pool_h2h3_fault_tests wedges a registered connection with sys:suspend/1 and fails if the pool waits on it. Before the change that scenario hit the eunit timeout; it now answers none in 252ms.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants