Narrow wildcard RBAC verbs to least-privilege on manager ClusterRole by abays · Pull Request #2027 · openstack-k8s-operators/openstack-operator

abays · 2026-08-11T10:25:59Z

Replace verbs: ["*"] with explicit minimal verb sets on the openstack-operator controller's kubebuilder RBAC markers and regenerated manifests:

admissionregistration.k8s.io webhooks: get/list/watch/create/update/delete/deletecollection
apiextensions.k8s.io: narrow resources from "*" to customresourcedefinitions
core serviceaccounts: get/list/delete (only used for cleanup)
core namespaces: get/create/update
core services: get/list/create/update/delete
Remove configmaps from init operator controller markers (unused; other controllers declare their own)

github-actions · 2026-08-11T10:26:19Z

OpenStackControlPlane CRD Size Report

Metric	Value
CRD JSON size	337535 bytes (330KB)
Base branch size	337535 bytes
Change	+0.00%
Status	yellow — growing

Threshold reference

Color	Range	Meaning
🟢 green	< 300KB	Comfortable
🟡 yellow	300–400KB	Growing
🟠 orange	400–750KB	Concerning
🔴 red	> 750KB	Approaching 1.5MB etcd limit (cut in half to allow space for update)

stuggi · 2026-08-13T09:45:46Z

https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/openstack-k8s-operators_openstack-operator/2027/pull-ci-openstack-k8s-operators-openstack-operator-main-openstack-operator-build-deploy-kuttl-4-20/2087123376488321024/artifacts/openstack-operator-build-deploy-kuttl-4-20/openstack-k8s-operators-gather/artifacts/must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-e951c596b1c06e42cc41260bfe9b929fa850a06bae1d25cda4a1289fdbebaf5b/namespaces/openstack-operators/pods/openstack-operator-controller-init-fc6c4fbf4-drmsh/logs/operator.log

go@v0.33.13/tools/cache/reflector.go:285", "type": "*v1.ServiceAccount", "error": "serviceaccounts is forbidden: User \"system:serviceaccount:openstack-operators:openstack-operator-controller-init\" cannot watch resource \"serviceaccounts\" in API group \"\" at the cluster scope"}
k8s.io/apimachinery/pkg/util/runtime.logError
	/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.33.13/pkg/util/runtime/runtime.go:226
k8s.io/apimachinery/pkg/util/runtime.handleError
	/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.33.13/pkg/util/runtime/runtime.go:217
k8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext
	/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.33.13/pkg/util/runtime/runtime.go:203
k8s.io/client-go/tools/cache.DefaultWatchErrorHandler
	/opt/app-root/src/go/pkg/mod/k8s.io/client-go@v0.33.13/tools/cache/reflector.go:200
k8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1
	/opt/app-root/src/go/pkg/mod/k8s.io/client-go@v0.33.13/tools/cache/reflector.go:360
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
	/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.33.13/pkg/util/wait/backoff.go:233
k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1
	/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.33.13/pkg/util/wait/backoff.go:255
k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext
	/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.33.13/pkg/util/wait/backoff.go:256
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
	/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.33.13/pkg/util/wait/backoff.go:233
k8s.io/client-go/tools/cache.(*Reflector).RunWithContext
	/opt/app-root/src/go/pkg/mod/k8s.io/client-go@v0.33.13/tools/cache/reflector.go:358
k8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3
	/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.33.13/pkg/util/wait/wait.go:63
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1
	/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.33.13/pkg/util/wait/wait.go:72

config/operator/rbac/role.yaml narrows serviceaccounts and services verbs to get,list,delete / get,list,create,update,delete — dropping watch. But deleteAllOwnedResources in openstack_controller.go:408,413 does typed List() calls (corev1.ServiceAccountList, corev1.ServiceList), which go through the controller-runtime cache and need watch for the informer's reflector. Without it, the manager will hit continuous Forbidden errors on Watch() and relist/backoff loops once that code path runs (triggered by OPENSTACK_RELEASE_VERSION changes). Not a hard break (List still works via periodic relist), but will spam errors for the life of the process, like above.

I think we either add the watch back, or switching those two List() calls to unstructured.

abays · 2026-08-13T10:32:13Z

https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/openstack-k8s-operators_openstack-operator/2027/pull-ci-openstack-k8s-operators-openstack-operator-main-openstack-operator-build-deploy-kuttl-4-20/2087123376488321024/artifacts/openstack-operator-build-deploy-kuttl-4-20/openstack-k8s-operators-gather/artifacts/must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-e951c596b1c06e42cc41260bfe9b929fa850a06bae1d25cda4a1289fdbebaf5b/namespaces/openstack-operators/pods/openstack-operator-controller-init-fc6c4fbf4-drmsh/logs/operator.log
go@v0.33.13/tools/cache/reflector.go:285", "type": "*v1.ServiceAccount", "error": "serviceaccounts is forbidden: User \"system:serviceaccount:openstack-operators:openstack-operator-controller-init\" cannot watch resource \"serviceaccounts\" in API group \"\" at the cluster scope"}
k8s.io/apimachinery/pkg/util/runtime.logError
	/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.33.13/pkg/util/runtime/runtime.go:226
k8s.io/apimachinery/pkg/util/runtime.handleError
	/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.33.13/pkg/util/runtime/runtime.go:217
k8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext
	/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.33.13/pkg/util/runtime/runtime.go:203
k8s.io/client-go/tools/cache.DefaultWatchErrorHandler
	/opt/app-root/src/go/pkg/mod/k8s.io/client-go@v0.33.13/tools/cache/reflector.go:200
k8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1
	/opt/app-root/src/go/pkg/mod/k8s.io/client-go@v0.33.13/tools/cache/reflector.go:360
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
	/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.33.13/pkg/util/wait/backoff.go:233
k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1
	/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.33.13/pkg/util/wait/backoff.go:255
k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext
	/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.33.13/pkg/util/wait/backoff.go:256
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
	/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.33.13/pkg/util/wait/backoff.go:233
k8s.io/client-go/tools/cache.(*Reflector).RunWithContext
	/opt/app-root/src/go/pkg/mod/k8s.io/client-go@v0.33.13/tools/cache/reflector.go:358
k8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3
	/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.33.13/pkg/util/wait/wait.go:63
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1
	/opt/app-root/src/go/pkg/mod/k8s.io/apimachinery@v0.33.13/pkg/util/wait/wait.go:72
config/operator/rbac/role.yaml narrows serviceaccounts and services verbs to get,list,delete / get,list,create,update,delete — dropping watch. But deleteAllOwnedResources in openstack_controller.go:408,413 does typed List() calls (corev1.ServiceAccountList, corev1.ServiceList), which go through the controller-runtime cache and need watch for the informer's reflector. Without it, the manager will hit continuous Forbidden errors on Watch() and relist/backoff loops once that code path runs (triggered by OPENSTACK_RELEASE_VERSION changes). Not a hard break (List still works via periodic relist), but will spam errors for the life of the process, like above.

I think we either add the watch back, or switching those two List() calls to unstructured.

Looks like we might need watch for Services as well, which I think you implied:

controller-runtime's cache-backed client creates an informer on-demand for any typed List() call, even for unregistered types. When deleteOwnedResources calls r.List(ctx, &corev1.ServiceList{}, ...) on line 414, the cache spins up a Service informer at that moment — same as  what happened with ServiceAccount. The services watch error just hasn't appeared yet because the code hits ServiceAccount first (line 409) and the reflector backoff keeps retrying that; once ServiceAccount resolves, the Service informer would hit the same wall on line 414.

centosinfra-prod-github-app · 2026-08-13T18:58:52Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/313aba5016294d4d9ef40afefe45231a

✔️ openstack-k8s-operators-content-provider SUCCESS in 4h 53m 17s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 37m 04s
❌ cifmw-crc-podified-edpm-baremetal MERGE_CONFLICT in 2m 18s
✔️ openstack-operator-tempest-multinode SUCCESS in 1h 56m 46s
✔️ openstack-operator-edpm-baremetal-minor-update SUCCESS in 2h 37m 05s

Replace verbs: ["*"] with explicit minimal verb sets on the openstack-operator controller's kubebuilder RBAC markers and regenerated manifests: - admissionregistration.k8s.io webhooks: get/list/watch/create/update/delete/deletecollection - apiextensions.k8s.io: narrow resources from "*" to customresourcedefinitions - core serviceaccounts: get/list/delete (only used for cleanup) - core namespaces: get/create/update - core services: get/list/create/update/delete - Remove configmaps from this controller's markers (unused; other controllers declare their own) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

centosinfra-prod-github-app · 2026-08-13T22:29:51Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/a52c1d65a35e4b2a93c05592d2033fb8

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 55m 37s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 34m 28s
❌ cifmw-crc-podified-edpm-baremetal FAILURE in 1h 17m 23s
✔️ openstack-operator-tempest-multinode SUCCESS in 1h 51m 28s
✔️ openstack-operator-edpm-baremetal-minor-update SUCCESS in 2h 27m 10s

stuggi · 2026-08-14T04:15:27Z

recheck

stuggi · 2026-08-14T06:34:40Z

/retest

centosinfra-prod-github-app · 2026-08-14T06:42:42Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/b35774e09a994cfbaa8ed3c7bfb03c69

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 25m 10s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 30m 53s
❌ cifmw-crc-podified-edpm-baremetal FAILURE in 35m 33s
✔️ openstack-operator-tempest-multinode SUCCESS in 1h 45m 19s
✔️ openstack-operator-edpm-baremetal-minor-update SUCCESS in 2h 10m 39s

abays · 2026-08-14T09:07:30Z

Build failed (check pipeline). Post recheck (without leading slash) to rerun all jobs. Make sure the failure cause has been resolved before you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/b35774e09a994cfbaa8ed3c7bfb03c69

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 25m 10s ✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 30m 53s ❌ cifmw-crc-podified-edpm-baremetal FAILURE in 35m 33s ✔️ openstack-operator-tempest-multinode SUCCESS in 1h 45m 19s ✔️ openstack-operator-edpm-baremetal-minor-update SUCCESS in 2h 10m 39s

Follow script's output here: /home/zuul/ci-framework-data/logs/ci_script_011_fetch_openshift_token.log
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: NoneType: None
task path: /home/zuul/src/github.com/openstack-k8s-operators/ci-framework/roles/openshift_login/tasks/try_login.yml:29
fatal: [localhost]: FAILED! => 
    changed: true
    msg: non-zero return code
    rc: 1
    stderr: ''
    stderr_lines: []
    stdout: |
      Unable to connect to the server: EOF
    stdout_lines:
    - 'Unable to connect to the server: EOF'

abays · 2026-08-14T09:07:37Z

recheck

abays · 2026-08-14T09:08:23Z

@abays: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:
Test name Commit Details Required Rerun command
ci/prow/openstack-operator-build-deploy-kuttl-4-20 3532763 link true /test openstack-operator-build-deploy-kuttl-4-20

Full PR test history. Your PR dashboard.
Details

No resources found in openstack-operators namespace.
No resources found in openstack-operators namespace.
No resources found in openstack-operators namespace.
No resources found in openstack-operators namespace.
make: *** [Makefile:899: openstack_wait] Error 124

/test openstack-operator-build-deploy-kuttl-4-20

stuggi

/lgtm

openshift-ci · 2026-08-17T13:09:56Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: abays, stuggi

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [abays,stuggi]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

stuggi · 2026-08-17T13:29:22Z

/test functional

abays · 2026-08-17T16:31:50Z

@abays: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:
Test name Commit Details Required Rerun command
ci/prow/openstack-operator-build-deploy-kuttl-4-20 3532763 link unknown /test openstack-operator-build-deploy-kuttl-4-20

Full PR test history. Your PR dashboard.
Details

    logger.go:42: 14:47:23 |  | error: unable to read URL "https://raw.githubusercontent.com/openstack-k8s-operators/infra-operator/main/config/samples/network_v1beta1_netconfig.yaml", server reported 429 Too Many Requests, status code=429
    logger.go:42: 14:47:23 |  | command failure, skipping 2 additional commands

/test openstack-operator-build-deploy-kuttl-4-20

abays · 2026-08-18T09:09:34Z

/cherry-pick 18-stable

openshift-cherrypick-robot · 2026-08-18T09:10:12Z

@abays: #2027 failed to apply on top of branch "18-stable":

Applying: Narrow wildcard RBAC verbs to least-privilege on manager ClusterRole
Using index info to reconstruct a base tree...
M	config/operator/rbac/role.yaml
M	config/rbac/role.yaml
M	internal/controller/operator/openstack_controller.go
Falling back to patching base and 3-way merge...
Auto-merging config/operator/rbac/role.yaml
Auto-merging config/rbac/role.yaml
Auto-merging internal/controller/operator/openstack_controller.go
CONFLICT (content): Merge conflict in internal/controller/operator/openstack_controller.go
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
hint: When you have resolved this problem, run "git am --continue".
hint: If you prefer to skip this patch, run "git am --skip" instead.
hint: To restore the original branch and stop patching, run "git am --abort".
hint: Disable this message with "git config set advice.mergeConflict false"
Patch failed at 0001 Narrow wildcard RBAC verbs to least-privilege on manager ClusterRole

Details

In response to this:

/cherry-pick 18-stable

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

abays requested review from dprince and stuggi August 11, 2026 10:25

openshift-ci Bot requested review from rebtoor and slagle August 11, 2026 10:26

openshift-ci Bot added the approved label Aug 11, 2026

abays force-pushed the narrow-wildcard-rbac-verbs branch 2 times, most recently from 247ac04 to 88c5cf8 Compare August 13, 2026 11:33

abays force-pushed the narrow-wildcard-rbac-verbs branch from 88c5cf8 to 3532763 Compare August 13, 2026 19:28

stuggi approved these changes Aug 17, 2026

View reviewed changes

openshift-ci Bot assigned stuggi Aug 17, 2026

openshift-ci Bot added the lgtm label Aug 17, 2026

openshift-merge-bot Bot merged commit fff4536 into openstack-k8s-operators:main Aug 17, 2026
8 checks passed

openstack-k8s-operators deleted a comment from openshift-cherrypick-robot Aug 18, 2026

Uh oh!

Conversation

abays commented Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

OpenStackControlPlane CRD Size Report

Uh oh!

stuggi commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

abays commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

centosinfra-prod-github-app Bot commented Aug 13, 2026

Uh oh!

centosinfra-prod-github-app Bot commented Aug 13, 2026

Uh oh!

stuggi commented Aug 14, 2026

Uh oh!

stuggi commented Aug 14, 2026

Uh oh!

centosinfra-prod-github-app Bot commented Aug 14, 2026

Uh oh!

abays commented Aug 14, 2026

Uh oh!

abays commented Aug 14, 2026

Uh oh!

abays commented Aug 14, 2026

Uh oh!

stuggi left a comment

Choose a reason for hiding this comment

Uh oh!

openshift-ci Bot commented Aug 17, 2026

Uh oh!

stuggi commented Aug 17, 2026

Uh oh!

abays commented Aug 17, 2026

Uh oh!

Uh oh!

abays commented Aug 18, 2026

Uh oh!

openshift-cherrypick-robot commented Aug 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

abays commented Aug 11, 2026 •

edited

Loading

github-actions Bot commented Aug 11, 2026 •

edited

Loading

stuggi commented Aug 13, 2026 •

edited

Loading

abays commented Aug 13, 2026 •

edited

Loading