feat: bootstrap new instances in-process instead of with Jobs by armru · Pull Request #11231 · cloudnative-pg/cloudnative-pg

armru · 2026-07-20T16:30:24Z

Bootstrapping an instance today goes through a dedicated batch/v1 Job per mode (<instance>-initdb, -join, -pgbasebackup, -full-recovery, -snapshot-recovery): the operator creates the PVCs, waits for the Job to initialize them, marks them ready, and only then creates the instance Pod. Every bootstrap therefore pays two pod schedules and image pulls, reconciliation stays blocked while the Job runs, and the operator has to coordinate two consumers of the same PVCs.

With this change the operator creates the instance Pod directly. When the pgdata PVC is not marked ready yet, the Pod carries a bootstrap overlay: instance run receives --bootstrap-mode=<mode> plus exactly the flags the Job builders used to emit, and the Pod is annotated with cnpg.io/bootstrapInstance for observability. Before starting the instance manager, instance run initializes PGDATA in-process with the same library code the instance <mode> subcommands are built on, while a minimal webserver keeps the kubelet probes and wal-restore working: the startup probe is answered during the whole data copy instead of timing out, readiness stays negative, and /pg/status answers 503 so the operator sees the instance as not reporting yet. The overlay is deliberately left out of the pod-template annotation, so a freshly bootstrapped Pod is not considered drifted and no rollout is triggered.

Completion is tracked at two levels. A marker file inside PGDATA records the mode and the instance identity (namespace, cluster name and UID, pod name): a container restarting mid-bootstrap finds no marker and retries the mode from scratch (except snapshot recovery, which resumes in place, since a snapshot-seeded PGDATA cannot be re-obtained), while a marker that traveled inside a volume snapshot belongs to a different identity and is ignored, so a snapshot-cloned PVC is bootstrapped properly instead of being mistaken for an already-initialized one. The cnpg.io/pvcStatus annotation remains the operator-side memory, and now flips to ready only when the owning Pod becomes Ready.

User-visible effects: no bootstrap Jobs are created anymore (the pg_upgrade Job is unchanged); bootstrap progress and failures surface on the instance Pod itself, in its logs and restart count; and since the Pod's hostname is the instance name from the start (the Jobs used the Job name), application_name and everything else derived from POD_NAME is correct during the initial data copy too. Recovery sources protected by a private CA no longer rely on a Secret mounted over /controller/certificates: the bootstrap writes the CA file itself, which also removes a mount that would have shadowed the directory where the instance manager maintains the server certificates.

Operator upgrades need no migration: the Job read/guard/cleanup paths are all kept, so an in-flight legacy bootstrap Job still blocks reconciliation, still marks its PVC ready on completion, and still surfaces its failure; from then on those paths only ever see the pg_upgrade Job.

Recovery through a CNPG-I plugin now runs inside the instance Pod, so plugins must serve the restore protocol from their instance sidecar (there is no restore Job carrying a dedicated sidecar anymore): cloudnative-pg/plugin-barman-cloud#1025 is the companion change. Removing the now-unused Job builders and the five bootstrap subcommands, and reworking how a failed bootstrap reaches PhaseUnrecoverable without a Job backoffLimit, are left to follow-ups; the failure policy is the open question tracked on the issue.

The e2e bootstrap assertions are retargeted from Jobs to Pods and now also verify that no bootstrap Job is ever created; a walStorage variant covers the separate WAL volume layout.

Closes #11228

github-actions · 2026-07-20T16:31:30Z

❗ By default, the pull request is configured to backport to all release branches.

To stop backporting this pr, remove the label: backport-requested ◀️ or add the label 'do not backport'
To stop backporting this pr to a certain release branch, remove the specific branch label: release-x.y

armru · 2026-07-20T17:07:57Z

/test barman_plugin=pr-1025

github-actions · 2026-07-20T17:08:42Z

@armru, here's the link to the E2E on CNPG workflow run: https://github.com/cloudnative-pg/cloudnative-pg/actions/runs/29762494135

armru · 2026-07-21T07:52:36Z

/test barman_plugin=pr-1025

github-actions · 2026-07-21T07:52:50Z

@armru, here's the link to the E2E on CNPG workflow run: https://github.com/cloudnative-pg/cloudnative-pg/actions/runs/29812087177

mnencia · 2026-07-31T16:34:56Z

/test

github-actions · 2026-07-31T16:35:26Z

@mnencia, here's the link to the E2E on CNPG workflow run: https://github.com/cloudnative-pg/cloudnative-pg/actions/runs/30647701073

Add a per-instance bootstrapProgress flag carrying the bootstrap mode and start time, so the instance manager can advertise that PostgreSQL is not up yet while it initializes the data directory in-process. While the flag is set the startup probe is skipped (the copy phase has no socket and can outlast the probe budget), /pg/status replies with a 503 and a recognizable body instead of a valid payload (a valid one would make the operator short-circuit its reconciliation guards on an unready pod), and an instance-manager update is refused with a 409. The flag is intentionally separate from mightBeUnavailable so a bootstrapping pod is never counted as reporting its status, which would break primary-before-replica ordering. Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>

Teach instance run to initialize an uninitialized PGDATA in-process before the controller-runtime manager starts, so a data directory can be bootstrapped without a dedicated Job. When run is given a --bootstrap-mode (plus the same per-mode flags the Job builders used to emit) and the data directory has no completion marker yet, run flips the instance bootstrap flag, serves a minimal status server and the local web server, dispatches the bootstrap through the shared library, and on success writes a durable completion marker inside PGDATA before falling through to the normal run. The manager cannot start earlier because its in-pod reconciler would write into an empty PGDATA, and the marker makes a container restart after a successful bootstrap skip straight to the normal run. The marker is fsynced together with its parent directory because WriteFileAtomic only flushes the file contents, not the rename. Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>

Prepare the operator to bootstrap an instance by turning its own pod into a bootstrapping pod instead of creating a dedicated Job. The per-mode flag lists that the six bootstrap Job builders assembled inline are extracted into shared arg builders, and a new ApplyBootstrapOverlay stamps those flags (prefixed with --bootstrap-mode) plus the volumes, mounts, environment and annotation each mode needs onto a steady-state instance pod produced by NewInstance. The overlay leaves the podSpec and podEnvHash annotations alone so a just-bootstrapped pod is not mistaken for a drifted one. Nothing calls the overlay yet, so the Job builders keep working exactly as before. Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>

The five bootstrap paths (initdb, join, pgbasebackup, recovery and volume-snapshot restore) no longer run in a dedicated batch Job before the instance Pod is created. The operator now builds the steady-state instance Pod and applies the bootstrap overlay to it, so the Pod initializes its own data directory in-process and then starts PostgreSQL. Primary creation, replica join and reattachment go through a single shared builder, and the lost-Job recovery that guarded #11036 becomes a plain Pod recreation for any serial whose data PVC never finished bootstrapping, covering both the interrupted first primary and a bootstrapping Pod evicted before it completed. A data PVC is now flipped to ready only once its owning Pod reports ready, instead of as soon as any Pod attaches, so a Pod still bootstrapping in-process no longer marks its not-yet-initialized volume as ready and cannot be mistaken for a fully bootstrapped one on the reattach path. The pg_upgrade Job and every Job read, guard and cleanup path are left untouched. Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>

…of mounting it The bootstrap overlay used to mount the barman endpoint CA Secret on the instance's certificates directory and point AWS_CA_BUNDLE at it. That was harmless for the short-lived bootstrap Job, but the job-free bootstrap keeps the same pod running as the instance, and a read-only Secret mount over that directory shadows the place where the instance manager writes its server certificate: the certificate reconciler then fails with a read-only filesystem error and the pod never becomes ready. Drop the CA mount from the overlay and instead write the recovery-source endpoint CA to disk during the in-process bootstrap, before barman-cloud runs, so the CA files are owned by the instance manager exactly as on every other pod. The write reuses the certificate reconciler's existing secret-to-file helper and resolves the source CA with the same precedence the Job builders used: the recovery backup reference, then the origin backup status, then the recovery source external cluster. The container-level AWS_CA_BUNDLE the Job injected is not restored, as the in-process restore and wal-restore build their own environment and only need the CA file present at the unchanged location. Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>

…identity The in-process phase-0 bootstrap writes a completion marker inside PGDATA so a container restart can skip work that already finished. Because the marker lives inside PGDATA, it travels inside volume snapshots: a PVC cloned from a snapshot already carries the source instance's marker, so IsCompleted reported the data directory as already bootstrapped and runBootstrap skipped the whole restoresnapshot phase-0. That lost the PITR recovery configuration (the cluster promoted from the snapshot data without fetching the archived WAL up to the requested target) and left snapshot cloned scale-up replicas without their demotion and replica configuration. Scope the marker to the instance that wrote it. The marker payload now records the namespace, cluster name, cluster UID and pod name, and IsCompleted reports completed only when the marker exists, parses, and matches the current instance. A marker inherited from another instance (or an unparseable or old-format one) is treated as not completed, so phase-0 runs and rewrites the marker with the new identity. The cluster is now loaded before the marker check to provide the cluster UID. Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>

Issue #11228 replaces every initdb/join/pgbasebackup/restore/restoresnapshot bootstrap Job with a self-bootstrapping instance Pod, so e2e assertions that checked batch/v1 Job objects for these paths no longer describe what the operator does. The microservice import-failure test now watches the bootstrap pod's restart count instead of a Job's failed count (the exact tier of the failure-surfacing policy is still open on #11228, so the assertion only relies on the pod restarting rather than the threshold); the #10985 regression test now checks that no new instance Pod appears for the recreated serial while the previous WAL PVC is terminating, instead of checking for the absence of a join Job. The initdb, join, pgbasebackup, and barman/volume-snapshot recovery suites (in-tree and via plugin-barman-cloud) gained a shared assertion that no bootstrap Job is ever created while a cluster is provisioned, plus a check that the resulting instance pods reach Ready without restarting their postgres container. A small tests/utils/jobs package and two new assertions in tests/internal/asserts/cluster (AssertNoBootstrapJobCreatedDuring, AssertClusterInstancesHaveNoRestart) back these checks so they can be reused across suites instead of inlining the same polling logic everywhere. Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>

The e2e list for #11228 had no walStorage variant: the bootstrap Job used to lay out pg_wal before run's ReconcileWalDirectory executed, and the in-process replacement needs the same ordering verified with a dedicated WAL volume configured. The existing dedicated-WAL-volume test already creates a 3-instance cluster with walStorage, exercising both initdb and join, so it is extended with the same no-bootstrap-Job and no-restart assertions used by the other bootstrap suites rather than adding a near-duplicate spec. Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>

…t CA secret Phase-0 bootstrap now reads the recovery endpoint CA through the Kubernetes API instead of the bootstrap Job's Secret volume mount, so the instance Role must list that secret among the ones it can get. The Role already enumerated the origin Backup credentials and the external cluster endpoint CA, but not the endpoint CA coming from the origin Backup status or from a recovery Backup reference in the cluster spec. Without those entries every restore from a Backup reference against a TLS-protected object store crashloops on a forbidden Secret get. Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>

With in-process bootstrap the operator reuses the freed instance serial and recreates the pod with the same name within seconds, so the old primary pod name never stays absent long enough for the test to observe it. Capture the old primary pod UID before deletion and assert the pod is either gone or has been replaced by a new one carrying a different UID, keeping the existing timeout. Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>

An in-process bootstrap now attempts exactly once. A bootstrap failure is deterministic (bad configuration, an unreachable or wrong recovery source, a corrupt backup), so re-running the same work does not recover: the instance records the failure in a marker kept beside PGDATA and parks instead of retrying, staying not ready and never contacting the recovery source again. Genuinely transient conditions are still retried inside the bootstrap step. The parked instance reports the failure on its status endpoint and the operator surfaces it as PhaseUnrecoverable, restoring the terminal signal the removed bootstrap Job used to provide through its backoff limit. Assisted-by: Claude <noreply@anthropic.com> Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>

Assisted-by: Claude <noreply@anthropic.com> Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>

The job-free import bootstrap now parks on a deterministic failure instead of crash-looping, so the nonexistent-source-database case is asserted via the cluster reaching PhaseUnrecoverable rather than the pod's restart count. Assisted-by: Claude <noreply@anthropic.com> Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>

A join or pgbasebackup clone reads from a live PostgreSQL server that can be transiently unavailable, for example while a primary restarts or switches over. Parking on the first failure turned such a transient error into a permanent PhaseUnrecoverable. Give these two modes a bounded retry (matching the backoff limit the bootstrap Job used to carry) and park only once it is exhausted; initdb, restore and snapshot restore still fail on the first attempt. Each retry re-runs the mode against a freshly emptied PGDATA. Assisted-by: Claude <noreply@anthropic.com> Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>

…thods handled by the overlay CreatePrimaryJobViaInitdb, CreatePrimaryJobViaRestoreSnapshot, CreatePrimaryJobViaRecovery, CreatePrimaryJobViaPgBaseBackup, JoinReplicaInstance and RestoreReplicaInstance built the Jobs used by the instance subcommands removed in the previous commit, so they and their dedicated jobRole values are dead code now that bootstrap goes through the pod overlay. CreatePrimaryJob is renamed to CreateRoleJob since it now only builds the generic role-tagged Job shell shared by the remaining callers (major upgrade). Signed-off-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com>

…gbasebackup instance subcommands These CLI entry points executed the primary/replica bootstrap Jobs that have been superseded by the pod overlay-based bootstrap, so they are no longer reachable code paths. Signed-off-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com>

…ailure marker Loading or recording the bootstrap failure marker only logged errors before, so a parked instance could report an empty or stale reason to the operator if the marker was missing or corrupted. FailBootstrap and LoadFailure now return errors, and parkFailedBootstrap propagates them instead of continuing, so the operator sees the inconsistency rather than a silent, reasonless park. Signed-off-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com>

leonardoce · 2026-08-06T14:58:03Z

/test bp=pr-1025 l=kind

github-actions · 2026-08-06T14:58:53Z

@leonardoce, here's the link to the E2E on CNPG workflow run: https://github.com/cloudnative-pg/cloudnative-pg/actions/runs/31113512127

leonardoce · 2026-08-07T06:51:19Z

/test bp=pr-1025 l=kind

github-actions · 2026-08-07T06:51:31Z

@leonardoce, here's the link to the E2E on CNPG workflow run: https://github.com/cloudnative-pg/cloudnative-pg/actions/runs/31155430935

…clone backoff Replace the hand-rolled attempt/select loop with retry.OnError and a Mode.RetryBackoff() helper, keeping the same retry tolerance for join/pgbasebackup while relying on a well-tested library primitive instead of custom loop bookkeeping. Signed-off-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com>

The in-process bootstrap marker identity did not include the volume it was written on, only the namespace, cluster, cluster UID and pod name. A replica re-created after a scale-down and scale-up that frees and reuses its node serial gets a PGDATA PVC cloned from a volume snapshot of the previous, same-named instance, and that clone already carries the predecessor's marker inside PGDATA. All four identity fields matched, so the inherited marker wrongly certified the clone as already bootstrapped and skipped the restoresnapshot phase-0 entirely, leaving the hot-snapshot leftovers of the live source instance in place (stale physical replication-slot state, temp and lock files) instead of running the restore cleanup, and voiding the marker's own guarantee that this instance bootstrapped this volume. The replica configuration itself is rewritten by the instance manager before PostgreSQL starts either way, so this did not leave the replica misconfigured; what it lost was the restore cleanup and any other phase-0 work. Record the PGDATA PVC UID in the marker identity, delivered to the instance as a new --bootstrap-pvc-uid overlay argument resolved from the PVC each caller already creates or reads. A snapshot-cloned PVC is a new object with a new UID, so the inherited marker stops matching and phase-0 runs again; a container restart on the same PVC, or the dangling-PVC reattach path, keeps matching since the PVC object itself is unchanged. Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>

HasFailed and LoadFailure both read and parsed the same failure marker file, so a parked instance re-read the marker a second time just to recover the mode and reason for the status endpoint, and that second read had to defensively handle the marker vanishing between the two calls even though nothing else touches it in between. Read the marker once with LoadFailure and pass the loaded failure straight into parkFailedBootstrap, dropping the second read and the now-impossible "vanished after being detected" branch it existed only to cover. Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>

joinReplicaInstance, recreateReplicaBootstrapPod and ensureInstanceBootstrapJob all set the owner reference, inherited the cluster metadata and created the Pod tolerating an AlreadyExists error from a stale cache, differing only in the Result they return once the Pod turns out to already exist. Move the shared part into createBootstrappingPod, which reports whether the Pod already existed and leaves the Result to each caller. Inline the clonesFromLiveServer predicate into RetryBackoff, its only caller, keeping the explanation of why a live-server clone is the only mode worth retrying in the RetryBackoff comment. Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>

BootstrapInstruction carried an addInitDBExtras boolean alongside the bootstrap mode, but the two always agreed: NewInitDBInstruction is the only constructor that sets the flag and also the only one that sets the initdb mode, so the single place reading the flag can read the mode instead. Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>

The phase-0 entry point already loads the Cluster to build the marker identity, so the helper that loads the server certificate for the status web server can take that object instead of fetching the same resource again from the API server. Two of the three bootstrapping Pod creation sites returned the same result whether or not the Pod was already there, so the boolean the creation helper reported is only telling callers something they do not act on. The helper now tolerates the AlreadyExists error on its own and returns nothing but an error. Finally, the recovery and restore-from-snapshot instructions never look at the Backup they receive: the operator resolves it while validating the recovery source, and the overlay is built from the Cluster alone. Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>

The two troubleshooting entries about a failing or hanging bootstrap still described it as an initialization Job. The hugepages crash and the unreachable API server are still the same problems with the same fixes, but they now happen inside the instance that bootstraps its own data directory, so the guidance points there. Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>

armru requested review from NiccoloFei, jsilvela and litaocdl as code owners July 20, 2026 16:30

armru requested a review from a team July 20, 2026 16:30

dosubot Bot added size:XXL This PR changes 1000+ lines, ignoring generated files. do not backport This PR must not be backported - it will be in the next minor release enhancement 🪄 New feature or request labels Jul 20, 2026

cnpg-bot added backport-requested ◀️ This pull request should be backported to all supported releases release-1.28 release-1.29 release-1.30 labels Jul 20, 2026

armru marked this pull request as draft July 20, 2026 16:39

dosubot Bot mentioned this pull request Jul 21, 2026

Cannot set the resources for initdb cloudnative-pg/charts#961

Closed

cnpg-bot added the ok to merge 👌 This PR can be merged label Jul 21, 2026

mnencia removed release-1.28 release-1.29 release-1.30 labels Jul 21, 2026

mnencia force-pushed the dev/remove-jobs branch from c59b44f to fe240d4 Compare July 21, 2026 15:43

mnencia force-pushed the dev/remove-jobs branch from fe240d4 to 84fde71 Compare July 31, 2026 06:49

mnencia removed the backport-requested ◀️ This pull request should be backported to all supported releases label Jul 31, 2026

mnencia marked this pull request as ready for review July 31, 2026 16:35

mnencia approved these changes Jul 31, 2026

View reviewed changes

dosubot Bot added the lgtm This PR has been approved by a maintainer label Jul 31, 2026

armru and others added 17 commits August 6, 2026 16:57

test: cover the bootstrap-failed status report

c0f7ecd

Assisted-by: Claude <noreply@anthropic.com> Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>

leonardoce force-pushed the dev/remove-jobs branch from f2fece5 to c6506c9 Compare August 6, 2026 14:57

leonardoce and others added 5 commits August 7, 2026 10:46

armru force-pushed the dev/remove-jobs branch from 5b92c0c to b88f56d Compare August 7, 2026 12:59

armru added 2 commits August 7, 2026 15:14

Navigation Menu

Uh oh!

feat: bootstrap new instances in-process instead of with Jobs - #11231

feat: bootstrap new instances in-process instead of with Jobs#11231
armru wants to merge 25 commits into
mainfrom
dev/remove-jobs

armru commented Jul 20, 2026

Uh oh!

github-actions Bot commented Jul 20, 2026

Uh oh!

armru commented Jul 20, 2026

Uh oh!

github-actions Bot commented Jul 20, 2026

Uh oh!

armru commented Jul 21, 2026

Uh oh!

github-actions Bot commented Jul 21, 2026

Uh oh!

mnencia commented Jul 31, 2026

Uh oh!

github-actions Bot commented Jul 31, 2026

Uh oh!

leonardoce commented Aug 6, 2026

Uh oh!

github-actions Bot commented Aug 6, 2026

Uh oh!

leonardoce commented Aug 7, 2026

Uh oh!

github-actions Bot commented Aug 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

armru commented Jul 20, 2026

Uh oh!

github-actions Bot commented Jul 20, 2026

Uh oh!

armru commented Jul 20, 2026

Uh oh!

github-actions Bot commented Jul 20, 2026

Uh oh!

armru commented Jul 21, 2026

Uh oh!

github-actions Bot commented Jul 21, 2026

Uh oh!

mnencia commented Jul 31, 2026

Uh oh!

github-actions Bot commented Jul 31, 2026

Uh oh!

leonardoce commented Aug 6, 2026

Uh oh!

github-actions Bot commented Aug 6, 2026

Uh oh!

leonardoce commented Aug 7, 2026

Uh oh!

github-actions Bot commented Aug 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants