rfc: evaluate coordinated rolling reboot mechanism for hypervisors

[adrianreza]

Background

Incident in Zendesk #369707 / RCA in #369908: the customer is asking for a commitment to either prevent recurrence through monitoring/automation or to adopt periodic fencing + reboot. Today there is no automated reboot mechanism in Atmosphere — unattended-upgrades is explicitly uninstalled in playbooks/kubernetes.yml (lines 51–56) — so any reboot is a manual operator activity.

Proposal

Evaluate a coordinated rolling-reboot mechanism for hypervisors, with the following requirements:

• Pre-drain hook that live-migrates instances off the host before reboot
• One host at a time per AZ/failure domain
• Configurable cadence (e.g. quarterly) and/or trigger from NodeUptimeExceeded
• Maintenance-window awareness
• Opt-in per environment

Candidates to evaluate: kured, custom Ansible playbook driven by AWX/Tower, or a small custom controller. Output of this issue is a design doc / decision; implementation tracked in a follow-up.

Acceptance criteria

• Comparison of candidate solutions
• Decision recorded (design doc or comment)
• Follow-up implementation issue opened
• If implemented: release note + admin docs

References

• Zendesk #369908 (RCA), #369707 (source incident)
• Companion issue: NodeUptimeExceeded alert (item Add monitoring for vni* interfaces #1 in the same RCA roadmap)