mgmt-hub: OpenBao auth plugin is staged in /tmp, so systemd-tmpfiles deletes it and the bao container fails to start (exit 127)

[joewxboy]

Summary

The all-in-one management hub stages the OpenBao auth plugin binary under /tmp and bind-mounts it into the bao container directly from there:

• mgmt-hub/deploy-mgmt-hub.sh:290 — TMP_DIR=/tmp/horizon-all-in-1

• mgmt-hub/deploy-mgmt-hub.sh:770-771 — plugin tarball extracted to $TMP_DIR/openbao/plugins

• mgmt-hub/docker-compose.yml:244:

  - /tmp/horizon-all-in-1/openbao/plugins/openbao-plugin-auth-openhorizon:/openbao/plugins/openbao-plugin-auth-openhorizon:ro

/tmp is not durable storage. On a stock Ubuntu host, /usr/lib/tmpfiles.d/tmp.conf contains:

D /tmp 1777 root root 30d

which means systemd-tmpfiles both wipes /tmp at every boot and deletes files older than 30 days during the daily systemd-tmpfiles-clean.timer run. The plugin binary is written once at install time and never touched again, so it reliably ages out after 30 days.

Why it fails silently, then breaks all at once

The running bao container holds the bind mount by inode, so it keeps working normally after the host file is deleted. Nothing is logged and nothing appears wrong — until the next time the container is recreated (Docker daemon upgrade/restart, host reboot, docker-compose up). At that point the mount source no longer exists.

Docker then auto-creates the missing bind source as a directory, while the container-side target is a file, producing a confusing error that does not mention the real problem:

failed to create task for container: failed to create shim task: OCI runtime create failed:
runc create failed: unable to start container process: error during container init:
error mounting "/tmp/horizon-all-in-1/openbao/plugins/openbao-plugin-auth-openhorizon" to rootfs at
"/openbao/plugins/openbao-plugin-auth-openhorizon": ... not a directory:
Are you trying to mount a directory onto a file (or vice-versa)?

The container exits 127. Note that although the service is declared restart: always, this is a create-time failure, so RestartCount stays at 0 and Docker never retries — the hub simply comes up permanently without OpenBao.

In our case the hub was installed 2026-05-23 and ran fine for two months. systemd-tmpfiles-clean.service last ran 2026-07-22, and a Docker daemon restart the following day left bao dead with the error above.

Steps to reproduce

1. Install the all-in-one management hub on a stock Ubuntu host:

   curl -sSL https://raw.githubusercontent.com/open-horizon/devops/master/mgmt-hub/deploy-mgmt-hub.sh -o deploy-mgmt-hub.sh
   chmod +x deploy-mgmt-hub.sh && ./deploy-mgmt-hub.sh

2. Confirm the plugin is staged in /tmp and the container is healthy:

   ls -l /tmp/horizon-all-in-1/openbao/plugins/openbao-plugin-auth-openhorizon
   docker ps --filter name=bao

3. Simulate the tmpfiles cleanup that occurs after 30 days (or a reboot):

   sudo rm -rf /tmp/horizon-all-in-1

   The bao container is still running and healthy at this point — this is the silent window.

4. Restart the container the way a Docker upgrade or reboot would:

   docker restart bao
   docker ps -a --filter name=bao      # Exited (127)
   docker inspect bao --format '{{.State.Error}}'

To observe it without waiting 30 days, you can instead confirm the aging rule directly with grep '^[dD] /tmp' /usr/lib/tmpfiles.d/tmp.conf and check systemctl status systemd-tmpfiles-clean.timer.

Suggested improvements

In rough order of preference:

1. Stage the plugin somewhere persistent. Anywhere outside /tmp fixes the root cause — e.g. a openbao/plugins/ directory next to the generated docker-compose.yml in the install directory, or /var/lib/horizon/openbao/plugins. This is a one-line change to TMP_DIR usage for this specific artifact plus the compose mount.

2. Or hold the plugin in a named Docker volume (e.g. bao-plugins-vol), populated at install time and mounted at /openbao/plugins. This puts the plugin under the same lifecycle management as bao-vol and bao-logs-vol, which are already named volumes, and keeps it consistent with how the rest of OpenBao's state is handled.

3. Mount the plugin directory rather than the individual file. Bind-mounting a single file is what turns a missing source into the misleading "directory onto a file" error. Mounting .../openbao/plugins → /openbao/plugins fails more clearly and also allows the plugin to be replaced/upgraded without editing the compose file.

4. Add a preflight check. Have deploy-mgmt-hub.sh verify the plugin binary exists before bringing up bao and re-stage it if not — ideally comparing against the SHA256 already registered in the OpenBao plugin catalog, since a mismatch there is its own failure mode.

5. If /tmp must be retained, ship a tmpfiles.d drop-in that exempts the path from cleanup and restores it at boot.

Workaround

For anyone hitting this now: re-download the plugin matching the version in OPENBAO_PLUGIN_AUTH_OPENHORIZON_VERSION (checksums are published on the plugin's release page), place it somewhere persistent, and restore it. The registered SHA256 in the catalog must match the binary, so the same release version must be used. The container then starts, and OpenBao needs unsealing again with the key from the install summary.

We additionally added an /etc/tmpfiles.d/ drop-in so the file is re-copied into /tmp at boot and exempted from age-based cleaning, which survives reboots without modifying the upstream compose file:

d /tmp/horizon-all-in-1/openbao/plugins 0755 root root -
C /tmp/horizon-all-in-1/openbao/plugins/openbao-plugin-auth-openhorizon 0755 root root - /path/to/persistent/openbao-plugin-auth-openhorizon
x /tmp/horizon-all-in-1

Environment

Deployment	deploy-mgmt-hub.sh all-in-one, defaults
OpenBao	2.0.3 (quay.io/openbao/openbao-ubi:2.0)
Plugin	openbao-plugin-auth-openhorizon v1.0.1
Host	Ubuntu 24.04 (kernel 6.8), Docker 29.6.2