Summary
In a production Atmosphere deployment, Horizon's Project -> Compute -> Instances page can return a 504 after about 180 seconds even for projects with a small number of instances. The direct slow dependency is Neutron port listing from Horizon: some Horizon code paths pass tenant_id into openstacksdk port list calls, which is dramatically slower than the equivalent project_id filter in this environment.
This appears to be a Horizon/openstacksdk filter compatibility/scaling issue rather than a Nova, Cinder, Glance, Horizon pod health, or project resource-count issue.
Environment
Observed deployment versions:
horizon image: ghcr.io/vexxhost/horizon:2025.1@sha256:35a079c3b9441190f1379d301d9c8ca84359c5457fa9ef209e1b8e3d9dcf7380
horizon: 25.3.2.dev5
openstacksdk: 4.4.0
python-novaclient: 18.9.0
python-neutronclient: 11.4.0
Django: 4.2.19
Relevant Horizon settings are at defaults/enabled in the deployment:
OPENSTACK_INSTANCE_RETRIEVE_IP_ADDRESSES = True
OPENSTACK_NEUTRON_NETWORK['enable_fip_topology_check'] = True
The deployment config only overrides SESSION_TIMEOUT; it does not override these Horizon Neutron settings.
Observed behavior
A fresh Horizon request to /project/instances/ was retried by ingress across all three Horizon pods. Each upstream hit the 60 second read timeout, and ingress returned a final 504 after about 180 seconds:
GET /project/instances/ -> 504 after ~180s
upstream timings: 60.001s, 60.005s, 60.002s
The Horizon pods later logged 200 for the same request, so the pods were not crashing. Horizon was completing the render after NGINX had already given up.
Backend evidence
During the same windows, Neutron logs showed broad list calls from Horizon pod IPs:
GET /v2.0/ports -> ~8.18 MB response, 32-36s
GET /v2.0/networks -> ~340 KB response, ~2-5s
GET /v2.0/subnets -> ~1.04 MB response, ~5-8s
Nova, Cinder, Glance, and project-scoped Neutron calls were small and fast. Example project-scoped calls for an affected project were on the order of tens to hundreds of milliseconds.
A direct timing test inside a Horizon pod showed the key difference:
project A:
ports(tenant_id=...) -> 47725ms, 7 ports
ports(project_id=...) -> 99ms, 7 ports
project B:
ports(tenant_id=...) -> 46964ms, 20 ports
ports(project_id=...) -> 242ms, 20 ports
Similar testing for networks showed tenant_id was also problematic in raw SDK usage, although Horizon's network_list() already normalizes tenant_id to project_id before calling openstacksdk.
Suspected cause
In the deployed Horizon code, openstack_dashboard/api/neutron.py normalizes tenant_id to project_id in network_list():
if 'tenant_id' in params:
params['project_id'] = params.pop('tenant_id')But port_list() does not perform the same normalization:
def port_list(request, **params):
ports = networkclient(request).ports(**params)FloatingIpManager uses this path with tenant_id:
port_search_opts = {'tenant_id': tenant_id}
ports = port_list(self.request, **port_search_opts)At this cloud scale, the tenant_id port-list path effectively behaves like a broad port scan, while project_id remains correctly scoped and fast.
Impact
This can make Horizon's instance page unusable for affected projects. Because the request goes through ingress retries, the user sees a long hang followed by a 504, even though the underlying OpenStack APIs are mostly healthy.
Proposed fix direction
Normalize tenant_id to project_id in Horizon's Neutron port-list wrapper before calling openstacksdk, matching the existing network_list() behavior:
if 'tenant_id' in params:
params['project_id'] = params.pop('tenant_id')Also audit other Horizon Neutron wrappers that pass tenant_id into openstacksdk list calls.
Mitigation notes
Increasing ingress/Horizon timeouts would only hide the symptom and keep the broad Neutron scans. A temporary mitigation may be possible by disabling high-cost Horizon Neutron/floating IP enrichment paths, but the durable fix should avoid the pathological tenant_id SDK filter path.
Summary
In a production Atmosphere deployment, Horizon's
Project -> Compute -> Instancespage can return a504after about 180 seconds even for projects with a small number of instances. The direct slow dependency is Neutron port listing from Horizon: some Horizon code paths passtenant_idinto openstacksdk port list calls, which is dramatically slower than the equivalentproject_idfilter in this environment.This appears to be a Horizon/openstacksdk filter compatibility/scaling issue rather than a Nova, Cinder, Glance, Horizon pod health, or project resource-count issue.
Environment
Observed deployment versions:
Relevant Horizon settings are at defaults/enabled in the deployment:
The deployment config only overrides
SESSION_TIMEOUT; it does not override these Horizon Neutron settings.Observed behavior
A fresh Horizon request to
/project/instances/was retried by ingress across all three Horizon pods. Each upstream hit the 60 second read timeout, and ingress returned a final504after about 180 seconds:The Horizon pods later logged
200for the same request, so the pods were not crashing. Horizon was completing the render after NGINX had already given up.Backend evidence
During the same windows, Neutron logs showed broad list calls from Horizon pod IPs:
Nova, Cinder, Glance, and project-scoped Neutron calls were small and fast. Example project-scoped calls for an affected project were on the order of tens to hundreds of milliseconds.
A direct timing test inside a Horizon pod showed the key difference:
Similar testing for networks showed
tenant_idwas also problematic in raw SDK usage, although Horizon'snetwork_list()already normalizestenant_idtoproject_idbefore calling openstacksdk.Suspected cause
In the deployed Horizon code,
openstack_dashboard/api/neutron.pynormalizestenant_idtoproject_idinnetwork_list():But
port_list()does not perform the same normalization:FloatingIpManageruses this path withtenant_id:At this cloud scale, the
tenant_idport-list path effectively behaves like a broad port scan, whileproject_idremains correctly scoped and fast.Impact
This can make Horizon's instance page unusable for affected projects. Because the request goes through ingress retries, the user sees a long hang followed by a
504, even though the underlying OpenStack APIs are mostly healthy.Proposed fix direction
Normalize
tenant_idtoproject_idin Horizon's Neutron port-list wrapper before calling openstacksdk, matching the existingnetwork_list()behavior:Also audit other Horizon Neutron wrappers that pass
tenant_idinto openstacksdk list calls.Mitigation notes
Increasing ingress/Horizon timeouts would only hide the symptom and keep the broad Neutron scans. A temporary mitigation may be possible by disabling high-cost Horizon Neutron/floating IP enrichment paths, but the durable fix should avoid the pathological
tenant_idSDK filter path.