pipeline-cmd: add --retries for transient connection errors

[tarzan]

What

Adds an opt-in --retries=N flag to pipeline-cmd that re-runs an app when its output matches a known transient connection error, with linearly increasing backoff (15s, 30s, …). Default is --retries=0 — existing behavior is byte-for-byte unchanged.

Why

ps:exec-style commands intermittently fail even when the dyno is up and running. Running an rpc across a ~200-app production stage surfaced three distinct flavors, all of which succeed on a plain re-run:

• Could not connect to dyno! — exec-manager rejects/times out the credential PUT (Establishing credentials... error)
• There was an error connecting to the dyno! — the SSH tunnel socket drops mid-session (ECONNRESET from the proxy)
• Connection to the dyno timed out! — missed keepalives

These show up disproportionately under parallel load — exactly how pipeline-cmd runs heroku — so today a big run reliably ends with a handful of apps whose record is a connection error rather than command output, and re-running the whole stage is the only recourse.

How

• A deliberately narrow TRANSIENT_CONNECTION_ERRORS pattern (the messages the heroku CLI's ps-exec code emits for connection-layer failures). Anything else is treated as a genuine command failure and is never retried.
• heroku_for_app_with_retries wraps heroku_for_app; the final attempt's output becomes the app's record either way, so a persistently failing app still surfaces its error. Each retry logs a notice to stderr so the data stream stays clean.
• HEROKU_SCRIPTS_RETRY_DELAY overrides the 15s backoff unit; the tests set it to 0 to stay fast.
• README documents the flag, including the caveat that a mid-session drop can occur after the remote command started, so --retries should only be used with idempotent commands.

Testing

• 5 new bats tests (retry-until-success, default single-attempt behavior preserved, persistent-error surfacing, no retry on genuine failures, flag validation) using call-counting stubs in the existing style.
• Full suite: 27/27 pass. shellcheck bin/heroku-scripts install.sh is clean (v0.11.0).

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Adds an opt-in retry mechanism to pipeline-cmd to automatically re-run per-app Heroku commands when the output indicates a known transient dyno connection failure, with linear backoff between attempts.

Changes:

• Introduces --retries=N to pipeline-cmd, plus HEROKU_SCRIPTS_RETRY_DELAY to control backoff timing.
• Implements retry logic via heroku_for_app_with_retries, retrying only when output matches a transient-connection error pattern.
• Adds Bats tests covering retry behavior, non-retry behavior for genuine failures, and flag validation; updates README/docs.

Reviewed changes

Copilot reviewed 2 out of 3 changed files in this pull request and generated no comments.

File	Description
bin/heroku-scripts	Adds --retries flag parsing/validation and retry wrapper around per-app Heroku execution.
test/heroku-scripts.bats	Adds tests validating retry-until-success, default behavior, persistent transient failures, non-retry failures, and flag validation.
README.md	Documents --retries usage, behavior, and the idempotency caveat.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.