Operator force-restarts a host that is still starting up: SYSTEM SHUTDOWN fails, scale-down fallback SIGTERMs it mid-load, and the aborted pass repeats forever

[ku524]

Summary

hostForceRestart() has no gate for "this host is still starting up". On a host whose ClickHouse has not begun listening on 8123 yet (slow metadata load), hostSoftwareRestart() fails immediately at the SYSTEM SHUTDOWN step (abort 2) and the operator escalates to hostScaleDown(), which scales the StatefulSet to 0 and SIGTERMs the host mid-load.

Because the reconcile pass then aborts, finalizeReconcileAndMarkCompleted() never advances the ancestor, so the next pass sees the same configuration diff, decides a restart is required again, and kills the host again. The host never gets enough uninterrupted time to finish starting. In our case this ran for hours and left a 3-replica cluster serving on a single replica, dropping async inserts.

Verified in release-0.26.3 (what we run) and still present in release-0.27.2 (line numbers below are from release-0.27.2).

Environment

• operator 0.26.3 (behavior re-verified against release-0.27.2 source)
• ClickHouse 26.3.17.56 and 26.3.12.3, EKS, EBS gp3 PVCs
• async_load_databases: 0 and async_load_system_database: 0, so metadata loads synchronously before the server starts listening. Load takes ~20-25 min for our table/part count. On top of that, kubelet's recursive fsGroup chown of the data volume (~3.9M files) delays container start by several more minutes.
• startupProbe is explicitly defined with a 3600s budget (failureThreshold: 360, periodSeconds: 10), so kubelet is content to wait. Only the operator is not.
• reconcile.statefulSet.update.timeout: 3900 (raised from the 900 we had, and from the 300 default)

Observed sequence

shouldForceRestartHost(): Config change(s) require host restart. Host: 0-1
reconcileHostStatefulSet(): Reconcile host STS force restart: 0-1
hostForceRestart(): Reconcile host. Force restart: 0-1
hostSoftwareRestart(): Host software restart start. Host: 0-1
schemer HostShutdown(): Host shutdown: 0-1
connect():FAILED Ping(http://clickhouse_operator:***@chi-clickhouse-default-0-1...:8123). Err: doRequest: transport...
Exec():FAILED connect(...) for SQL: SYSTEM SHUTDOWN
retry: exec(): FAILED single try. No retries will be made for Applying sqls
hostSoftwareRestart(): Host software restart abort 2. Host: 0-1 err: FAILED connect(...)
hostScaleDown(): Reconcile host. Host shutdown via scale down: 0-1
Poll(): delete StatefulSet: clickhouse/chi-clickhouse-default-0-1: WAIT

Kubernetes side: SuccessfulDelete Delete Pod chi-clickhouse-default-0-1-0 in StatefulSet ... successful, container terminated by SIGTERM while still loading. The next reconcile pass repeats the same sequence.

The pod's startupProbe had not failed. kubelet was still within its budget. The host was killed purely by the operator.

Why the existing knobs do not prevent it

• reconcile.host.wait.probes.startup: yes does not help. It is consumed in prepareStsReconcileOptsWaitSection(), which runs after the force-restart block in reconcileHostStatefulSet():

  // pkg/controller/chi/worker-reconciler-chi.go
  if w.shouldForceRestartHost(ctx, host) {
      _ = w.hostForceRestart(ctx, host, opts)      // <-- host is killed here
  }
  w.stsReconciler.PrepareHostStatefulSetWithStatus(...)
  opts = w.prepareStsReconcileOptsWaitSection(host, opts)   // <-- probes are read here

• reconcile.statefulSet.update.timeout does not help either. The waits it bounds (waitHostIsStarted / waitHostIsRunning / waitHostIsReady, aborts 4/5/6) are only reached after HostShutdown() succeeds. A still-starting host fails at abort 2 and never gets there.

• shouldForceRestartHost() (pkg/controller/chi/worker.go:153) checks stopped / troubleshoot / new / no-ancestor / image-change / IsRollingUpdate() / IsConfigurationChangeRequiresReboot() / crashed-with-unknown-version. None of these express "still starting".

• spec.suspend stops the killing, but it stops all reconciliation, so it cannot be part of normal operation.

Why it becomes a loop

finalizeReconcileAndMarkCompleted() (pkg/controller/chi/worker.go:396) is the only place that advances the ancestor:

if util.IsContextDone(ctx) {
    log.V(1).Info("Reconcile is aborted. cr: %s ", _cr.GetName())
    return
}
...
c.SetAncestor(c.GetTarget())

Any aborted pass leaves status.normalizedCompleted stale, so IsConfigurationChangeRequiresReboot() keeps returning true for the same settings diff on every subsequent pass, and every pass force-restarts the host again. Killing a starting host guarantees the pass aborts, which guarantees the next pass repeats it.

Once the pod ends up in CrashLoopBackOff, the host.Runtime.Version.IsUnknown() && w.isPodCrushed(ctx, host) case adds a second reason to force-restart, further entrenching the loop.

Suggested fix

The needed predicate already exists and does not block:

// pkg/controller/chi/worker-status-helpers.go:145
func (w *worker) isPodStarted(ctx context.Context, host *api.Host) bool {
	if pod, err := w.c.kube.Pod().Get(ctx, host); err == nil {
		return k8s.PodHasAllContainersStarted(pod)
	}
	return false
}

Adding a case to shouldForceRestartHost() (or an early return in hostForceRestart()) that defers the restart while the pod has not passed its startup probe would be sufficient:

case !w.isPodStarted(ctx, host):
    // The host cannot answer SQL yet, so a software restart is impossible and the
    // scale-down fallback would restart the metadata load from scratch.
    return false

Hosts that never start are still handled: kubelet enforces startupProbe, and a genuinely crash-looping pod is already covered by the isPodCrushed case.

Alternatively, distinguishing "connection refused because the server is not up yet" from other HostShutdown() failures inside hostSoftwareRestart(), and not escalating to hostScaleDown() in that case, would achieve the same.

Two smaller things noticed in the same area:

1. pkg/model/chi/creator/probe.go returns a liveness-shaped probe for the default startup probe:

   case interfaces.ProbeDefaultStartup:
       return m.createDefaultLivenessProbe(host)

   pkg/model/chk/creator/probe.go has a proper createDefaultStartupProbe. With reconcile.host.wait.probes.startup: yes and no explicit startupProbe, a CHI host therefore gets a ~90s budget (initialDelay 60, period 3, failureThreshold 10), which kubelet then enforces against a slow-starting host.

2. reconcile.recovery.onStatus.completed.onPodNotReady (new in 0.27.2) pushes in the opposite direction for this scenario: a host that is loading is NotReady, so enabling it would scale the host down sooner.

Related

• Operator does not wait for table part load before restarting other replicas #1682 (open) — operator does not wait for part load before restarting other replicas. Same underlying assumption about fast startup.
• clickhouse db pod restarts all the time #1691 (closed without a code change) — same log signature (connect():FAILED Ping(...:8123), Exec():FAILED connect(...) for SQL:), reporter's diagnosis was also "the operator does not wait for the port to open".
• StatefulSet restart fallback can report success while host StatefulSet and pod are missing #1995 / PR Fix StatefulSet recreate failure handling #1997 — same hostSoftwareRestart → hostScaleDown fallback, but scoped to it reporting success while the StatefulSet and pod are gone.
• Image upgrade with version-specific settings causes CrashLoopBackOff due to ConfigMap update before StatefulSet reconciliation #1926 — SYSTEM SHUTDOWN issued before the StatefulSet rollout, producing old image + new config.

Happy to send a PR for the isPodStarted gate if that direction looks right.