Nova: missing connection pool config for [api_database] and [cell0_database] causes recurring QueuePool exhaustion

[adrianreza]

Summary

Nova API pods repeatedly experience sqlalchemy.exc.TimeoutError: QueuePool limit of size 5 overflow 50 reached, connection timed out, timeout 30.00 against the nova_api and nova_cell0 databases. This causes recurring API outages (5xx responses, instances launching in ERROR state).

The root cause is that the Nova helm values rendered by Atmosphere only configure connection pool tuning for the [database] section — the [api_database] and [cell0_database] sections are missing pool settings, so they fall back to oslo.db defaults (connection_recycle_time=3600s, no max_pool_size/max_overflow consistent with [database]).

Observed behavior

On a single active PXC backend (HAProxy active/standby setup):

• 1,237 Nova connections accumulated (nova: 681, nova_api: 317, nova_cell0: 239)
• Oldest idle connection: 4.2 hours
• Nova API QueuePool exhausted within 5 workers × 50 overflow per pod
• Errors continue to recur (latest occurrence today, 2026-04-24 14:01 UTC)

The rendered nova.conf shows:

[database]
connection_recycle_time = 600
max_pool_size = 5
max_overflow = 50
pool_timeout = 30
max_retries = -1

[api_database]
max_retries = -1
# missing: connection_recycle_time, max_pool_size, max_overflow, pool_timeout

[cell0_database]
max_retries = -1
# missing: connection_recycle_time, max_pool_size, max_overflow, pool_timeout

Root cause

In internal/openstack_helm/nova.go, the NovaConf struct only defines a single Database *DatabaseConf field:

type NovaConf struct {
    Database *DatabaseConf
    // ApiDatabase  *DatabaseConf  <-- missing
    // Cell0Database *DatabaseConf <-- missing
}

As a result, only [database] receives the pool config from internal/openstack_helm/database.go (ConnectionRecycleTime, MaxPoolSize, MaxRetries). The [api_database] and [cell0_database] sections render with only the defaults set elsewhere in the chart (e.g. max_retries: -1 from charts/nova/values.yaml).

The validation test in internal/testutils/oslo_db.go (TestDatabaseConf) only checks the [database] section, so this gap was not caught.

Impact

• nova_api and nova_cell0 connections never recycle on the configured 600s interval — they live until oslo.db's 3600s default or until the DB drops them
• Connection pile-up on the active PXC backend (HAProxy active/standby) → exhaustion of MySQL Threads_connected capacity and thread/memory pressure
• Customers see HTTP 500s with sqlalchemy.exc.TimeoutError

Suggested fix

1. Add ApiDatabase *DatabaseConf and Cell0Database *DatabaseConf fields to NovaConf in internal/openstack_helm/nova.go.
2. Render them into [api_database] and [cell0_database] in the Nova helm values, using the same defaults as [database] (or allow per-section overrides).
3. Extend TestDatabaseConf in internal/testutils/oslo_db.go to validate [api_database] and [cell0_database] sections as well.
4. (Optional) Default manifests.cron_job_archive_deleted_rows: true so deleted-row buildup doesn't compound the connection pressure on the same DB.

References

• internal/openstack_helm/nova.go — NovaConf struct missing fields
• internal/openstack_helm/database.go — DatabaseConf struct (template to reuse)
• internal/testutils/oslo_db.go — TestDatabaseConf only validates [database]
• charts/nova/values.yaml — [api_database]/[cell0_database] only set max_retries: -1
• oslo.db connection recycling: https://docs.openstack.org/oslo.db/latest/reference/opts.html