fix: mcc infra run silently swallows docker compose failures - #60
Open
dogfootman wants to merge 1 commit into
Open
fix: mcc infra run silently swallows docker compose failures#60dogfootman wants to merge 1 commit into
dogfootman wants to merge 1 commit into
Conversation
common.SysCall discarded docker compose up's exit code, so mcc infra run always returned 0 even when Compose aborted the startup graph partway through (e.g. a service failing its healthcheck). installAll.sh could never detect this, leaving services stuck in Created state with no error reported. - run.go now uses RunE + SysCallWithError so the real exit code propagates, and sets SilenceUsage to avoid dumping help text on runtime failures. - installAll.sh brings the stack up in staged waves instead of one monolithic call, checks each wave's exit code immediately, and points to iam_manager_init.sh if IAM setup didn't complete. - monitor_containers now uses `docker ps -a` so Created/Exited containers are classified as failures instead of being indistinguishable from "still pulling", and stops polling immediately instead of waiting out the full 20-minute timeout.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
common.SysCalldiscardeddocker compose up's exit code, somcc infra runalways returned 0 even when Compose aborted the startup graph partway through (e.g. a service failing its healthcheck).installAll.shcould never detect this, leaving unrelated services stuck inCreatedstate with no error reported andmc-iam-manager-post-initialsilently never running.run.gonow usesRunE+common.SysCallWithError(already used elsewhere in the codebase) so the real exit code propagates, and setsSilenceUsageto avoid dumping help text on runtime failures.installAll.shbrings the stack up in staged waves (core infra → data/web → observability backbone → app tier) instead of one monolithic call, checks each wave's exit code immediately and names the failing wave/service, and points to the existingiam_manager_init.shif IAM setup didn't complete.monitor_containersnow usesdocker ps -asoCreated/Exitedcontainers are classified as failures instead of being indistinguishable from "still pulling", and stops polling immediately instead of waiting out the full 20-minute timeout.docker/run.go(the commandinstallAll.shdepends on) — the samecommon.SysCallpattern exists instop/remove/pull/update/info.goand thek8spackage, left alone sincecleanAll.shrelies onstop/removetolerating "already stopped/removed" as non-fatal.