feat(monitoring): add ovn-controller stall and disconnect alerts

[adrianreza]

Background

Incident in Zendesk #369707 / RCA in #369908: ovn-controller exhibited poll-loop stalls of up to 54s on hypervisors with leaked OVS ports, causing OVS disconnects after the 5s inactivity probe and packet loss for VMs on the affected hosts. There is currently no alerting on ovn-controller health.

Related existing issue: #401 (Open vSwitch hangs but still passes readiness tests).

Proposal

Add Prometheus alerts for ovn-controller health, e.g.:

• OVNControllerPollLoopStalled — long poll-loop intervals
• OVNControllerSouthboundDisconnected — controller lost SBDB connection
• OVNControllerCPUSaturated — sustained high CPU on the controller process

This likely requires scraping ovn-controller's Unix-socket stats (e.g. via a small textfile-collector script or a dedicated exporter). The exporter/scrape mechanism should be designed first, then the rules built on top.

Acceptance criteria

• Decide on metric source (textfile collector vs exporter)
• Scrape mechanism deployed via Atmosphere
• Alert rules with promtool tests
• Runbook entries
• Release note

References

• Zendesk #369908 (RCA), #369707 (source incident)
• Open vSwitch hangs but still passes readiness tests #401