feat: add globalresourcequota api - #2068
Open
oliverbaehler wants to merge 15 commits into
Open
Conversation
Signed-off-by: Oliver Baehler <oliver@sudo-i.net>
Signed-off-by: Oliver Baehler <oliver@sudo-i.net>
Signed-off-by: Oliver Baehler <oliver@sudo-i.net>
…paths Signed-off-by: Oliver Baehler <oliver@sudo-i.net>
…paths Signed-off-by: Oliver Baehler <oliver@sudo-i.net>
…paths Signed-off-by: Oliver Baehler <oliver@sudo-i.net>
…paths Signed-off-by: Oliver Baehler <oliver@sudo-i.net>
…paths Signed-off-by: Oliver Baehler <oliver@sudo-i.net>
…paths Signed-off-by: Oliver Baehler <oliver@sudo-i.net>
…paths Signed-off-by: Oliver Baehler <oliver@sudo-i.net>
Signed-off-by: Oliver Baehler <oliver@sudo-i.net>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces a new GlobalResourceQuota API and end-to-end enforcement flow that applies a shared Kubernetes ResourceQuotaSpec across multiple namespaces, including support for tenant rule–generated global quotas plus admission-time usage calculations and reservation coordination via QuantityLedger.
Changes:
- Add
GlobalResourceQuotaCRD/API types, controller reconciliation, ledger coordination, and Prometheus metrics/alerts. - Add admission-time calculation/enforcement webhook (plus a Kubernetes-quota-derived evaluator) to prevent oversubscription across namespaces.
- Extend Tenant rules with a
quotasection that generatesGlobalResourceQuotaobjects, and ensure quota fields aren’t propagated into RuleStatus/enforcement caches.
Reviewed changes
Copilot reviewed 52 out of 54 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| pkg/tenant/rules.go | Excludes quota definitions from per-namespace RuleStatus body projection. |
| pkg/tenant/rules_promotions_test.go | Adds coverage ensuring quota is not projected into namespace rule status. |
| pkg/tenant/rule_quota.go | Implements generation helpers for rule-derived GlobalResourceQuota objects. |
| pkg/tenant/rule_quota_test.go | Tests deterministic naming and selector/quota generation for rule-derived quotas. |
| pkg/tenant/metdata.go | Removes staticcheck suppressions around deprecated regex fields (needs adjustment). |
| pkg/ruleengine/validate.go | Adds quota rule validation (name format, uniqueness, hard constraints). |
| pkg/ruleengine/validate_test.go | Adds tests for quota validation cases. |
| pkg/api/rules/zz_generated.deepcopy.go | Adds deepcopy support for quota rule types and fields. |
| pkg/api/rules/rule_body_types.go | Extends namespace rule body with quota list. |
| pkg/api/rules/resource_quota_types.go | Adds ResourceQuotaRule type for rule-defined quotas. |
| pkg/api/meta/metadata.go | Marks new managed labels for global quotas/rule quota tracking. |
| pkg/api/meta/labels.go | Adds label keys for GlobalResourceQuota and rule quota identity. |
| internal/webhook/tenant/validation/rule_validator.go | Validates tenant rules that include quota-only entries and GRQ name constraints. |
| internal/webhook/route/globalresourcequota.go | Adds webhook routing for global quota admission calculations. |
| internal/webhook/globalresourcequota/calculation.go | Implements admission-time validation/enforcement with ledger reservations. |
| internal/webhook/globalresourcequota/calculation_test.go | Adds tests for atomic reservations, concurrency, and formatting. |
| internal/quota/evaluator/evaluator.go | Adds Kubernetes-derived usage evaluator for admission requests. |
| internal/quota/evaluator/evaluator_test.go | Adds evaluator tests for compute, object counts, scopes, and constraints. |
| internal/metrics/global_resourcequota_recorder.go | Adds Prometheus metrics recorder for GRQ readiness/usage. |
| internal/metrics/global_resourcequota_recorder_test.go | Tests GRQ metrics recording and cleanup. |
| internal/controllers/tls/utils.go | Adds the GlobalResourceQuota CRD to managed CRDs list. |
| internal/controllers/tenant/manager.go | Watches GlobalResourceQuota objects owned by Tenants; syncs rule GRQs. |
| internal/controllers/tenant/globalresourcequotas.go | Adds rule→GRQ reconciliation and pruning logic. |
| internal/controllers/tenant/globalresourcequotas_test.go | Tests GRQ generation stability and pruning for tenant rules. |
| internal/controllers/rulestatus/manager.go | Ensures RuleStatus reconciliation strips quota definitions; cleans legacy status. |
| internal/controllers/rulestatus/manager_test.go | Adds tests ensuring quota is excluded and legacy status is cleaned. |
| internal/controllers/globalresourcequotas/manager.go | Wires the GlobalResourceQuota controller into the manager. |
| internal/controllers/globalresourcequotas/ledger.go | Adds ledger ensure/reconcile logic for GRQ reservation coordination. |
| internal/controllers/globalresourcequotas/controller.go | Implements GRQ controller: namespace selection, child RQ sync, usage observation. |
| internal/controllers/globalresourcequotas/controller_test.go | Tests usage observation, ledger reconciliation, and selector semantics. |
| hack/distro/capsule/example-setup/tenants.yaml | Updates example Tenant manifests to use rule-based quota definitions. |
| hack/distro/capsule/example-setup/kustomization.yaml | Includes GlobalResourceQuota example manifest in kustomization. |
| hack/distro/capsule/example-setup/global-resource-quotas.yaml | Adds example GlobalResourceQuota manifest. |
| go.sum | Adds dependency checksums for new Kubernetes helper modules. |
| go.mod | Adds k8s.io/component-helpers dependency for evaluator logic. |
| e2e/suite_test.go | Adds cleanup for GRQs labeled env=e2e after suite completion. |
| e2e/rules_quota_test.go | Adds E2E coverage for rule-generated GRQs and enforcement behavior. |
| e2e/global_resource_quota_test.go | Adds E2E coverage for direct GRQ objects and enforcement behavior. |
| cmd/controller/main.go | Registers GRQ controller and calculation webhook route/handler. |
| charts/capsule/values.yaml | Adds webhook config + alert rules for GlobalResourceQuota metrics. |
| charts/capsule/values.schema.json | Adds schema for the new globalresourcequotas webhook config. |
| charts/capsule/templates/rbac.yaml | Grants controller RBAC for GlobalResourceQuota resources. |
| charts/capsule/templates/hooks/crd-lifecycle/rbac.yaml | Grants CRD lifecycle hook RBAC for the new CRD. |
| charts/capsule/templates/configuration.yaml | Renders validating webhook config for GRQ calculations. |
| charts/capsule/README.md | Documents new Helm values for globalresourcequotas webhook. |
| charts/capsule/crds/capsule.clastix.io_tenants.yaml | Updates Tenant CRD schema to include rule quota definitions. |
| charts/capsule/crds/capsule.clastix.io_rulestatuses.yaml | Updates RuleStatus CRD schema for quota fields (even if excluded from status). |
| charts/capsule/crds/capsule.clastix.io_quantityledgers.yaml | Extends QuantityLedger CRD schema with ResourceQuota coordination state. |
| charts/capsule/crds/capsule.clastix.io_globalresourcequotas.yaml | Adds GlobalResourceQuota CRD definition. |
| api/v1beta2/zz_generated.deepcopy.go | Adds deepcopy implementations for GRQ and ledger quota status types. |
| api/v1beta2/quantityledgers_status.go | Adds ResourceQuota reservation/status types to QuantityLedger status. |
| api/v1beta2/globalresourcequota_types.go | Adds GlobalResourceQuota API type definitions. |
| api/v1beta2/globalresourcequota_status.go | Adds GRQ status fields for namespaces, totals, and conditions. |
| api/v1beta2/globalresourcequota_func.go | Adds GRQ helper funcs (names, ledger name, available calc, etc.). |
Files not reviewed (1)
- api/v1beta2/zz_generated.deepcopy.go: Generated file
Suppressed comments (1)
pkg/tenant/metdata.go:192
sc.Regexaccesses the deprecatedAllowedListSpec.Regexfield (see pkg/api/allowed_list.go). Removing the previous//nolint:staticcheckwill likely reintroduce staticcheck failures while legacy regex annotations are still supported.
Comment on lines
180
to
182
| if len(ic.Regex) > 0 { | ||
| annotations[meta.AvailableIngressClassesRegexpAnnotation] = ic.Regex | ||
| } |
Signed-off-by: Oliver Baehler <oliver@sudo-i.net>
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 53 out of 55 changed files in this pull request and generated no new comments.
Files not reviewed (1)
- api/v1beta2/zz_generated.deepcopy.go: Generated file
Suppressed comments (2)
internal/webhook/globalresourcequota/calculation.go:369
- An explicitly empty namespace selector (e.g.
namespaceSelectors: [{}]) will decode withLabelSelector == nil(because it's an inlined pointer) and is currently skipped, meaning the quota matches no namespaces. The API docs say an empty selector matches all namespaces; treatLabelSelector == nilaslabels.Everything()/ match-all.
internal/controllers/globalresourcequotas/controller.go:126 - A
namespaceSelectors: [{}]entry decodes withLabelSelector == nil(inlined pointer), whichselectors.GetNamespacesMatchingSelectorscurrently skips, resulting in zero selected namespaces. Normalize nil selectors to an explicit emptyLabelSelector{}before calling into the selectors helper so "empty selector matches all namespaces" works as documented.
namespaces, err := selectors.GetNamespacesMatchingSelectors(
ctx,
r.reader,
instance.Spec.NamespaceSelectors,
)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.