Skip to content

Reconcile SQL hook queries fire only on the first host; subsequent hosts silently no-op #2052

Description

@rahghoo

Any SQL hook defined under spec.reconcile.host.hooks.pre executes correctly on the first target in a reconcile, but silently no-ops on every subsequent target. No error surfaces and the operator logs the hook as firing, but the queries slice has been zeroed out in place, so nothing reaches ClickHouse.

Steps to reproduce

1. Apply a minimal CHI with 2 hosts and a host-shutdown SQL hook

cat <<'EOF' | kubectl apply -f -
apiVersion: clickhouse.altinity.com/v1
kind: ClickHouseInstallation
metadata:
  name: repro
  namespace: default
spec:
  configuration:
    clusters:
      - name: c1
        layout:
          shardsCount: 1
          replicasCount: 2
  reconcile:
    host:
      hooks:
        pre:
          - sql:
              queries:
                - SYSTEM STOP FETCHES
                - SYSTEM STOP MERGES
                - SYSTEM STOP REPLICATION QUEUES
                - SYSTEM STOP DISTRIBUTED SENDS
            events: [HostShutdown]
            failurePolicy: Ignore
EOF

2. Trigger a rolling restart

Any pod-template change works. I bumped terminationGracePeriodSeconds by 1 via helm upgrade. Both hosts get restarted sequentially by the operator.

Actual result

Only host 0-0 receives the queries. The operator log for 0-1 shows an empty payload:

Running SQL host hook on 0-0: [SYSTEM STOP FETCHES SYSTEM STOP MERGES SYSTEM STOP REPLICATION QUEUES SYSTEM STOP DISTRIBUTED SENDS]
Running SQL host hook on 0-1: [   ]

Expected behaviour

Both hosts receive the four SYSTEM STOP … queries before shutdown. The operator log shows the same populated queries list for both hosts:

Running SQL host hook on 0-0: [SYSTEM STOP FETCHES SYSTEM STOP MERGES SYSTEM STOP REPLICATION QUEUES SYSTEM STOP DISTRIBUTED SENDS]
Running SQL host hook on 0-1: [SYSTEM STOP FETCHES SYSTEM STOP MERGES SYSTEM STOP REPLICATION QUEUES SYSTEM STOP DISTRIBUTED SENDS]

Environment

  • Altinity ClickHouse Operator: 0.27.1
  • ClickHouse: 26.7.1

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions