Skip to content

feat(neutron): detect and clean up orphaned OVS br-int ports #3893

Description

Background

Incident in Zendesk #369707 / RCA in #369908: on long-uptime hypervisors with high VM churn (CI workloads), OVS leaks br-int ports when libvirt removes the tap interface but the corresponding OVS port entry is not removed. One affected host had 1,634 ports on br-int, of which 1,591 were orphaned. This is what drove the ovn-controller poll-loop stalls.

We currently have neutron-netns-cleanup-cron (see charts/neutron/templates/daemonset-netns-cleanup-cron.yaml) but it does not touch OVS ports.

Proposal

Two parts:

  1. Detection — Prometheus alert on excessive port count on br-int and/or on the orphan ratio (ports without a corresponding live tap).
  2. Cleanup — DaemonSet/cron (modeled after neutron-netns-cleanup-cron) that periodically diffs ovs-vsctl list-ports br-int against live libvirt domain interfaces and removes orphans, with conservative safety checks.

Acceptance criteria

  • Cleanup script/DaemonSet with dry-run mode and metrics
  • Alert rule(s) with promtool tests
  • Runbook
  • Release note
  • Documented opt-out for environments not using libvirt+OVS

References

  • Zendesk #369908 (RCA), #369707 (source incident)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions