feat(monitoring): alert on excessive host uptime (NodeUptimeExceeded)

[adrianreza]

Background

Incident in Zendesk #369707 / RCA in #369908: hypervisors with 800+ days of uptime developed Mellanox NIC degradation (RX buffer exhaustion, TX pause frames) that was only resolved by a reboot. We had no alert telling us those hosts were dangerously overdue for maintenance.

Proposal

Add a Prometheus alert (e.g. NodeUptimeExceeded) based on node_boot_time_seconds, firing at:

• P4 after 180 days uptime
• P3 after 365 days uptime

Implement as a new rule in roles/kube_prometheus_stack/files/jsonnet/ with promtool tests and a runbook entry under doc/source/admin/monitoring.rst.

Acceptance criteria

• Alert defined with severity labels matching existing conventions
• promtool tests in roles/kube_prometheus_stack/files/jsonnet/tests.yml
• Runbook section with diagnostic + remediation guidance (drain → live-migrate → reboot)
• Release note via reno new

References

• Zendesk ticket #369908 (RCA)
• Zendesk ticket #369707 (source incident)