Background
Incident in Zendesk #369707 / RCA in #369908: on long-uptime hypervisors with high VM churn (CI workloads), OVS leaks br-int ports when libvirt removes the tap interface but the corresponding OVS port entry is not removed. One affected host had 1,634 ports on br-int, of which 1,591 were orphaned. This is what drove the ovn-controller poll-loop stalls.
We currently have neutron-netns-cleanup-cron (see charts/neutron/templates/daemonset-netns-cleanup-cron.yaml) but it does not touch OVS ports.
Proposal
Two parts:
- Detection — Prometheus alert on excessive port count on
br-int and/or on the orphan ratio (ports without a corresponding live tap).
- Cleanup — DaemonSet/cron (modeled after
neutron-netns-cleanup-cron) that periodically diffs ovs-vsctl list-ports br-int against live libvirt domain interfaces and removes orphans, with conservative safety checks.
Acceptance criteria
References
- Zendesk #369908 (RCA), #369707 (source incident)
Background
Incident in Zendesk #369707 / RCA in #369908: on long-uptime hypervisors with high VM churn (CI workloads), OVS leaks
br-intports when libvirt removes the tap interface but the corresponding OVS port entry is not removed. One affected host had 1,634 ports onbr-int, of which 1,591 were orphaned. This is what drove the ovn-controller poll-loop stalls.We currently have
neutron-netns-cleanup-cron(seecharts/neutron/templates/daemonset-netns-cleanup-cron.yaml) but it does not touch OVS ports.Proposal
Two parts:
br-intand/or on the orphan ratio (ports without a corresponding live tap).neutron-netns-cleanup-cron) that periodically diffsovs-vsctl list-ports br-intagainst live libvirt domain interfaces and removes orphans, with conservative safety checks.Acceptance criteria
References