[Magnum] OCCM can silently skip cloud-node and retain stale Nodes without provider IDs

[adrianreza]

Summary

A Magnum/Cluster API Kubernetes cluster deployed by Atmosphere entered a repeated worker-remediation loop:

1. A replacement VM booted successfully and completed kubeadm join.
2. The Kubernetes Node remained without an OpenStack providerID and internal address.
3. The Node retained node.cloudprovider.kubernetes.io/uninitialized.
4. MachineHealthCheck replaced the Machine after nodeStartupTimeout.
5. The cycle repeated.

The OCCM DaemonSet pods were Running and Ready, but the elected leader was not running the configured cloud-node controller. Restarting only the elected leader caused cloud-node-controller to start, after which the current replacement Node was initialized and the Machine reached Running.

A large number of stale Node objects had also accumulated. These Nodes had no provider ID and no corresponding CAPI Machine or Nova server.

This report intentionally excludes customer, environment, cluster, host, project, IP, ticket, timestamp, and workload identifiers.

Environment observed

• Atmosphere: 7.2.0
• Magnum Cluster API: 0.36.6
• Kubernetes: 1.34.x
• OCCM chart: openstack-cloud-controller-manager-2.35.0
• OCCM image: registry.k8s.io/provider-os/openstack-cloud-controller-manager:v1.34.1
• OCCM deployment: three-replica DaemonSet on control-plane Nodes with leader election
• Add-on delivery: per-cluster ClusterResourceSet using Reconcile

Symptoms

The lifecycle controller repeatedly logged the generic error below for Nodes whose provider ID was empty and whose Nova server no longer existed:

error checking if node <node-name> exists: failed to find object

The process health endpoint continued reporting healthy even though cloud-node was absent from the active controller set. New Nodes therefore joined at the kubelet level but were never initialized by the cloud provider.

Findings

1. Missing-instance errors are not translated for name lookup

In cloud-provider-openstack v1.34.1, InstancesV2.getInstance() returns getServerByName() directly when node.Spec.ProviderID is empty.

When no matching server exists, that path returns raw errors.ErrNotFound. However, InstanceExists() only interprets cloudprovider.InstanceNotFound as a deleted instance. The raw error prevents the node lifecycle controller from removing the stale Node.

This is tracked upstream in kubernetes/cloud-provider-openstack#3114 and is also reported against OCCM v1.35.0, so a minor image upgrade alone is not currently sufficient.

2. Configured cloud controllers can fail to start without failing the process

Kubernetes v1.34.1's shared cloud-controller startup code handles a NewCloudNodeController() error by logging a warning and returning:

return nil, false, nil

The same pattern exists for cloud-node-lifecycle. This allows the controller-manager process and health endpoint to remain healthy after an essential configured controller was skipped.

The observed controller set and recovery after leader restart are consistent with this path.

3. The existing Magnum override is not specific enough for a patched image

Magnum Cluster API supports cloud_provider_tag, but it only changes the tag. The image repository remains the upstream registry.

The available container_infra_prefix setting changes the registry for other cluster images as well, which is broader than necessary for safely deploying a patched OCCM build.

Proposed path for Atmosphere

1. Carry a minimal downstream OCCM patch until fixed upstream releases are available:

   • Translate name-lookup ErrNotFound to cloudprovider.InstanceNotFound.
   • Add a regression test for an empty provider ID with no matching Nova server.
   • Make failure to start configured cloud-node and cloud-node-lifecycle controllers fatal/observable.

2. Publish patched OCCM images per supported Kubernetes minor using immutable internal digests.

3. Add a dedicated full-image override to Magnum Cluster API, for example cloud_provider_image, instead of reusing the global container registry prefix.

4. Render the override through the existing reconciled ClusterResourceSet so existing clusters roll safely and new clusters inherit the fix from their templates.

5. Add supplementary OCCM process probes and monitoring for:

   • Nodes retaining node.cloudprovider.kubernetes.io/uninitialized.
   • Nodes missing provider IDs beyond a short initialization window.
   • Repeated MachineHealthCheck remediation.
   • Differences between active CAPI Machine, Nova server, and Kubernetes Node counts.

Process probes alone do not detect a controller that was skipped during startup, so fail-fast behavior or controller-specific monitoring is required.

Acceptance criteria

• A Node with an empty provider ID and no matching Nova server is reported as not existing without an error and is removed by the lifecycle controller.
• Failure to initialize a configured cloud controller makes OCCM restart or become unready instead of silently continuing.
• A replacement worker receives its provider ID and internal IP before MachineHealthCheck's startup timeout.
• Magnum can select a complete OCCM image reference or digest without redirecting unrelated Kubernetes images.
• Updating the image rolls OCCM one control-plane Node at a time and can be reverted through the same reconciled source.

References

• Node is not deleted from the cluster, after being deleted from openstack kubernetes/cloud-provider-openstack#3114
• https://github.com/kubernetes/cloud-provider-openstack/blob/v1.34.1/pkg/openstack/instances.go
• https://github.com/kubernetes/kubernetes/blob/v1.34.1/staging/src/k8s.io/cloud-provider/app/core.go
• https://github.com/vexxhost/magnum-cluster-api/blob/v0.36.6/src/magnum.rs
• https://github.com/vexxhost/magnum-cluster-api/blob/v0.36.6/src/addons/cloud_controller_manager.rs