From a23877f5ac7348910f793a1fd3a7e81212453fb5 Mon Sep 17 00:00:00 2001 From: Oliver Baehler Date: Thu, 23 Jul 2026 14:33:39 +0200 Subject: [PATCH 1/5] fix: nav links Signed-off-by: Oliver Baehler --- content/en/docs/quickstart/_index.md | 37 ++++++++++++++++++++-------- 1 file changed, 27 insertions(+), 10 deletions(-) diff --git a/content/en/docs/quickstart/_index.md b/content/en/docs/quickstart/_index.md index f180430..7d471d9 100644 --- a/content/en/docs/quickstart/_index.md +++ b/content/en/docs/quickstart/_index.md @@ -73,7 +73,9 @@ Perspective of the Cluster Administrator, who is responsible for creating and ma In Capsule, a Tenant is an abstraction to group multiple namespaces in a single entity within a set of boundaries defined by the Cluster Administrator. -#### [Ownership](/docs/tenants/permissions/#ownership) +#### Ownership + +[Read More](/docs/tenants/permissions/#ownership) The tenant is then assigned to a user or group of users who is called [`TenantOwner`](/docs/operating/architecture/#tenant-owners). Capsule defines a Tenant as Custom Resource with cluster scope. Create the tenant as cluster admin: @@ -213,7 +215,9 @@ spec: quota: 2 ``` -#### [Prefix](/docs/tenants/administration/#force-tenant-prefix) +#### Prefix + +[Read More](/docs/tenants/administration/#force-tenant-prefix) We are enforcing the `Namespaces` of the `Tenant` to be prefixed with the `Tenant` name. This keeps the sorting of `Namespaces` clean and directly tells us which `Tenant` a `Namespace` belongs to. This is done with the `forceTenantPrefix` option in the `Tenant` spec: @@ -236,11 +240,15 @@ spec: forceTenantPrefix: true ``` -### [Rules](/docs/tenants/rules/) +### Rules + +[Read More](/docs/tenants/rules/) With [Rules](/docs/tenants/rules/) we can apply different policies within a `Tenant` based on their metadata. As previously seen they can also be used to enforce metadata for `Namespaces`. This comes in handy when we have different applications environment in the same `Tenant` and we want to apply different policies to them. For example, we can have a `Tenant` with two namespaces: `solar-production` (`environment=prod`) and `solar-development` (`environment=dev`) . We can apply different rules to each namespace based on their metadata. -#### [Metadata](/docs/rules/enforcement/metadata/) +#### Metadata + +[Read More](/docs/rules/enforcement/metadata/) Since the `Namespaces` are managed by the `TenantOwners`, we may want to require certain metadata to be present in the namespaces created within a `Tenant`. For this case we want to force the [`TenantOwners`](#tenant-owners) to provide the label `environment` with a value of either `prod`, `test` or `dev` when creating a namespace within the `solar` tenant. This can be done with [namespace metadata](/docs/tenants/metadata/#requiredmetadata): @@ -454,7 +462,9 @@ spec: managed: "restricted" ``` -#### [Permissions](/docs/tenants/rules/permissions/) +#### Permissions + +[Read More](/docs/tenants/rules/permissions/) Often you may have other users with different permissions. These are not [Tenant Owners](/docs/operating/architecture/#tenant-owners) but might be other parties that may interact with the `Tenant` and its `Namespaces`. For example, we may have a group of users that are responsible for monitoring the `Tenant` and its `Namespaces`. We can create a set of rules to allow them to view the `Tenant` and its `Namespaces` but not modify them. This can be done with [permissions rules](/docs/tenants/rules/permissions/): @@ -499,7 +509,9 @@ spec: name: tenant:{{ .tenant.metadata.name }}:operators ``` -#### [Workloads](/docs/rules/enforcement/workloads/) +#### Workloads + +[Read More](/docs/rules/enforcement/workloads/) There might also be different requirements for the priority of workloads running in different namespaces. For example, we may want to allow `BestEffort` Pods in the `solar-development` namespace but not in the `solar-production` namespace. This can be done with [Workload Rules](/docs/rules/enforcement/workloads/#best-effort): @@ -586,7 +598,9 @@ spec: env: "production" ``` -#### [Services](/docs/rules/enforcement/services/) +#### Services + +[Read More](/docs/rules/enforcement/services/) Often from a platform perspective, we want to control the type of services that can be created within a `Tenant`. It is possible to restrict the type of services that can be created within a `Tenant` with [Service Rules](/docs/rules/enforcement/services/#service-types). For example, we can allow only `ClusterIP` services: @@ -646,7 +660,9 @@ spec: - exp: ".*\\.{{ .tenant.metadata.name }}\\.svc\\.company\\.com" ``` -### [Resource Quota](/docs/tenants/quotas/) +### Resource Quota + +[Read More](/docs/tenants/quotas/) Another improtant aspect of the `Tenant` is the ability to define a set of [`ResourceQuotas`](https://kubernetes.io/docs/concepts/policy/resource-quotas/) for the entire `Tenant`. This allows the Cluster Administrator to control the amount of resources that can be used by the `Tenant` and its namespaces. For example, we can define a resource quota for the entire `Tenant`: @@ -818,7 +834,9 @@ NAME STATE NAMESPACE QUOTA NAMESPACE COUNT NODE SELECTOR READY STA solar Active 2 0 True reconciled 35s ``` -### [Replications](/docs/replications/) +### Replications + +[Read More](/docs/replications/) From a platform perspective, we may want to enforce certain objects per `Namespace` of `Tenant's`. With Replications we can enforce certain objects to be present in all `Namespaces` of a `Tenant`. See the following examples for common use cases of [replications](/docs/replications/). @@ -826,7 +844,6 @@ From a platform perspective, we may want to enforce certain objects per `Namespa Distribute a [`NetworkPolicy`](https://kubernetes.io/docs/concepts/services-networking/network-policies/) to all `Namespaces` of a `Tenant` to enforce a certain network policy for all workloads within the `Tenant`/`Namespace`. The following `NetworkPolicy` is an attempt to achieve a default deny policy for all `Namespaces` of the `Tenant` but allow intra-namespace communication and allow communication between all `Namespaces` of the same `Tenant`. It also allows communication to system namespaces (eg. monitoring, ingress, etc.). [Read More](https://kubernetes.io/docs/concepts/security/multi-tenancy/#network-isolation) - [Get Here](/docs/quickstart/gtr-netpol.yaml) ```yaml From 092bf7436883ff362761efb69590d5f0255496de Mon Sep 17 00:00:00 2001 From: Oliver Baehler Date: Thu, 23 Jul 2026 15:09:14 +0200 Subject: [PATCH 2/5] fix: nav links Signed-off-by: Oliver Baehler --- content/en/docs/rules/enforcement/metadata.md | 71 +++++++++++++++---- 1 file changed, 56 insertions(+), 15 deletions(-) diff --git a/content/en/docs/rules/enforcement/metadata.md b/content/en/docs/rules/enforcement/metadata.md index e93512a..f57c4dc 100644 --- a/content/en/docs/rules/enforcement/metadata.md +++ b/content/en/docs/rules/enforcement/metadata.md @@ -65,8 +65,7 @@ Capsule-managed labels include labels used to track Tenant ownership, resource p Because these keys are owned by Capsule, metadata rules that reference them are ignored by default. Use application-specific labels and annotations for Tenant policy enforcement. - -### Target resources +## Target resources Each metadata rule defines which resource kinds it applies to: @@ -122,7 +121,7 @@ metadata: This can match resources such as `apps/v1` `ReplicaSet` and `apps/v1` `StatefulSet`. -#### Namespace +### Namespace `Namespace` is the only cluster-scoped resource supported by metadata rules. It is deliberately opt-in: the `kinds` list must contain the literal, @@ -184,7 +183,7 @@ metadata: In short, both conditions must be true: `apiGroups` must match core `v1`, and `kinds` must contain a dedicated `Namespace` entry. -#### Important `apiGroups` behavior +### Important `apiGroups` behavior Omitted or empty `apiGroups` does **not** mean all API groups and versions. It means the core Kubernetes API version `v1`. @@ -220,7 +219,7 @@ metadata: - Deployment ``` -### Label rules +## Label rules Label rules are configured under `metadata[].labels`. Each map key is the label key to validate. @@ -285,7 +284,7 @@ Example rejection: Error from server (Forbidden): error when creating "configmap.yaml": admission webhook "rules.generic.projectcapsule.dev" denied the request: metadata label "env" is required at metadata.labels["env"] ``` -### Annotation rules +## Annotation rules Annotation rules are configured under `metadata[].annotations`. Each map key is the annotation key to validate. @@ -361,7 +360,49 @@ data: key: value ``` -### Required metadata +## Default + +The `default` field provides a value for coressponding field should no value be provided by the user. This is only applied at admission time and does not enforce the value to be present in the object. + +`default` is meaningful for `action: allow`. `deny` and `audit` rules are value matchers; they do not require missing metadata to exist. + +```yaml +rules: + - enforce: + action: allow + metadata: + - kinds: + - ConfigMap + labels: + cost-center: + default: "internal" +``` + +Default values still validate against the configured `values` matchers. If the default value does not match any allow or deny rule, the request is denied. + +## Managed + +Providing managed values ensures the metadata is always set to the provided value. This is applied at admission time and also enforced by the `RuleStatus` controller. Meaning it's also applied to already existing objects and also enforced at admission time. This is useful for enforcing certain metadata to be present and also to ensure the value is always set to a specific value. + +```yaml +rules: + - enforce: + action: allow + metadata: + - apiGroups: + - "v1" + kinds: + - Namespace + annotations: + example.corp/cost-center: + required: false + values: + - exp: "^INV-[0-9]{4}$" + # Overwrites anything, even if the user has set a value, Should be applied using SSA by the rulestatus controller, if removed also removes (one fieldmanager per rulestatus which controlles all managed metadata). Also enforce at admission + managed: "INV-10" +``` + +## Required The `required` field controls whether the metadata key must be present. @@ -401,7 +442,7 @@ data: If the label is missing, the request is denied. -### Metadata values +## Validation The `values` field uses the common match expression structure with `exact`, `exp`, and optional `negate`. @@ -451,9 +492,9 @@ With this rule: If an allow-list also exists for the same metadata key, values excluded from a negated deny rule still need a matching allow rule. -### Advanced +## Advanced -#### Allow-list behavior for metadata +### Allow-list behavior for metadata An `allow` rule creates an allow-list for the specific metadata key it controls. @@ -535,7 +576,7 @@ rules: The object must contain both `env=prod` and `team=platform`. -#### Deny metadata values +### Deny metadata values Use `action: deny` to reject specific metadata values. @@ -599,7 +640,7 @@ rules: In namespaces labeled `allow-deprecated=true`, `environment=deprecated` is admitted because the later namespace-specific allow rule matches. -#### Audit metadata values +### Audit metadata values Use `action: audit` to observe metadata usage without blocking the request. @@ -623,7 +664,7 @@ A matching object is admitted in this audit-only example, but Capsule emits an a If an allow-list also exists for the same metadata key, audit does not satisfy that allow-list. The metadata value must still match an `allow` rule. -#### Multiple resource kinds +### Multiple resource kinds A single metadata rule can target multiple kinds: @@ -648,7 +689,7 @@ rules: With this rule, both matching `ConfigMap` and `Service` objects must contain `corp.com/tenant=prod` or `corp.com/tenant=test`. -#### Namespace-specific metadata rules +### Namespace-specific metadata rules Metadata enforcement supports `namespaceSelector` like other namespace rules. @@ -672,7 +713,7 @@ rules: This rule only applies to namespaces labeled `environment=prod`. In those namespaces, matching `ConfigMap` objects must contain `example.corp/approval=approved`. -#### Complete metadata enforcement example +### Complete metadata enforcement example The following example combines required labels, optional annotations, multiple kinds, audit rules, deny rules, and namespace-specific exceptions: From 8a0c835ed5cd4d4986e61d8c5e98f6965fe5af15 Mon Sep 17 00:00:00 2001 From: Oliver Baehler Date: Thu, 23 Jul 2026 15:12:14 +0200 Subject: [PATCH 3/5] fix: nav links Signed-off-by: Oliver Baehler --- content/en/docs/rules/enforcement/metadata.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/content/en/docs/rules/enforcement/metadata.md b/content/en/docs/rules/enforcement/metadata.md index f57c4dc..c5a6b9f 100644 --- a/content/en/docs/rules/enforcement/metadata.md +++ b/content/en/docs/rules/enforcement/metadata.md @@ -311,10 +311,8 @@ rules: required: false values: - exp: "^INV-[0-9]{4}$" - # If user / annotation is missing, use this as default value, only at admission mutation - default: "II-1" # Overwrites anything, even if the user has set a value, Should be applied using SSA by the rulestatus controller, if removed also removes (one fieldmanager per rulestatus which controlles all managed metadata). Also enforce at admission - managed: "II-10" + managed: "INV-10" example.corp/cost-center-2: values: - exp: "II-10" From 93b7442f637a8733fe2a4732b78adc17b0730a38 Mon Sep 17 00:00:00 2001 From: Oliver Baehler Date: Thu, 23 Jul 2026 15:28:59 +0200 Subject: [PATCH 4/5] fix: nav links Signed-off-by: Oliver Baehler --- content/en/docs/quickstart/_index.md | 59 ++++++++++++++++++++++++++++ 1 file changed, 59 insertions(+) diff --git a/content/en/docs/quickstart/_index.md b/content/en/docs/quickstart/_index.md index 7d471d9..5855e00 100644 --- a/content/en/docs/quickstart/_index.md +++ b/content/en/docs/quickstart/_index.md @@ -970,6 +970,65 @@ spec: type: Container ``` +#### Showcase: Tenant Scope + +[Explore](/docs/replications/global/#examples) + +They key difference in the example is, that we use `scope: Tenant` instead of `scope: Namespace`. This creates Items for each `Tenant`, not for each `Namespace` of a `Tenant`. This allows us to create a single SopsProvider for the entire Tenant and distribute secrets across all namespaces of the Tenant. In this example we will showcase how to use the [Sops Operator](https://github.com/peak-scale/sops-operator) to distribute secrets across all namespaces of a tenant and for each `Tenant` we provide [`GlobalProxySettings`](/docs/proxy/proxysettings/#globalproxysettings). It also showcases more advanced templating machinsms to generate resources based on the `Tenant` metadata and status. The `GlobalTenantResource` is a powerful tool to manage resources across all namespaces of a tenant. + +```yaml +--- +apiVersion: capsule.clastix.io/v1beta2 +kind: GlobalTenantResource +metadata: + name: tenant-sops-providers +spec: + resyncPeriod: 600s + scope: Tenant + resources: + - generators: + - missingKey: zero + template: | + --- + apiVersion: capsule.clastix.io/v1beta1 + kind: GlobalProxySettings + metadata: + name: {{ $.tenant.metadata.name }}-proxy-settings + spec: + rules: + - subjects: + {{- range $.tenant.status.owners }} + - kind: {{ .kind }} + name: {{ .name }} + {{- end }} + clusterResources: + - apiGroups: + - "capsule.clastix.io" + resources: + - "globalcustomquotas" + operations: + - List + selector: + matchLabels: + company.com/tenant: {{ $.tenant.metadata.name }} + - rawItems: + - apiVersion: addons.projectcapsule.dev/v1alpha1 + kind: SopsProvider + metadata: + name: "{{tenant.name}}" + spec: + keys: + - namespaceSelector: + matchLabels: + capsule.clastix.io/tenant: "{{tenant.name}}" + sops: + - namespaceSelector: + matchLabels: + capsule.clastix.io/tenant: "{{tenant.name}}" +``` + + + ## Tenant Owners Each tenant comes with a delegated user or group of users acting as the tenant admin. In the Capsule jargon, this is called the [`TenantOwner`s](/docs/operating/architecture/#tenant-owners). Other users can operate inside a tenant with different levels of permissions and authorizations assigned directly by the `TenantOwner`. From 2863ae75dce2fe25ce5a3301b479227552611d36 Mon Sep 17 00:00:00 2001 From: Oliver Baehler Date: Fri, 31 Jul 2026 10:35:27 +0200 Subject: [PATCH 5/5] feat: add grq example Signed-off-by: Oliver Baehler --- .../customquotas/_index.md | 2 +- .../globalresourcequota/_index.md | 565 ++++++++++++++++++ .../resourcepools/_index.md | 2 +- 3 files changed, 567 insertions(+), 2 deletions(-) create mode 100644 content/en/docs/resource-management/globalresourcequota/_index.md diff --git a/content/en/docs/resource-management/customquotas/_index.md b/content/en/docs/resource-management/customquotas/_index.md index b8f447f..54e7689 100644 --- a/content/en/docs/resource-management/customquotas/_index.md +++ b/content/en/docs/resource-management/customquotas/_index.md @@ -1,6 +1,6 @@ --- title: Custom Quotas -weight: 6 +weight: 3 description: > CustomQuotas let you define and enforce arbitrary, label-scoped limits for any Kubernetes resource kind or CRD, at namespace or cluster scope. --- diff --git a/content/en/docs/resource-management/globalresourcequota/_index.md b/content/en/docs/resource-management/globalresourcequota/_index.md new file mode 100644 index 0000000..d2dc072 --- /dev/null +++ b/content/en/docs/resource-management/globalresourcequota/_index.md @@ -0,0 +1,565 @@ +--- +title: Global Resource Quotas +description: Share one atomic Kubernetes resource budget across multiple namespaces. +weight: 1 +--- + +`GlobalResourceQuota` provides a single Kubernetes `ResourceQuota` budget shared by a selected set of namespaces. + +A standard Kubernetes `ResourceQuota` is namespaced. Copying the same quota into multiple namespaces gives every namespace the full limit, so the combined usage can grow with the number of namespaces. `GlobalResourceQuota` keeps the familiar Kubernetes quota API while enforcing one aggregate hard limit across all selected namespaces. + +The resource is cluster-scoped and can select any namespace. Selected namespaces do not have to belong to a Capsule Tenant. + +## Concept + +A `GlobalResourceQuota` consists of: + +- One or more namespace label selectors. +- One native Kubernetes `ResourceQuotaSpec`. +- Aggregate usage and availability in its status. +- Per-namespace usage in its status. +- An internal `QuantityLedger` used to serialize concurrent admission. + +Capsule creates a managed native `ResourceQuota` in every selected namespace. The native quotas provide Kubernetes-compatible accounting and status. Capsule's validating admission webhook evaluates all matching `GlobalResourceQuota` objects and reserves new usage atomically before allowing the request. + +This combination prevents concurrent requests in different namespaces from exceeding the shared limit. + +### Benefits + +- Share one hard limit across any number of namespaces. +- Use native Kubernetes resource names, scopes, and scope selectors. +- Select Tenant namespaces, application namespaces, environments, or arbitrary namespace groups. +- Prevent oversubscription during concurrent admission. +- Inspect aggregate and per-namespace usage from one cluster-scoped resource. +- Monitor limits, usage, availability, conditions, and namespace consumption with Prometheus. +- Generate quotas from Capsule Tenant rules where desired. + +## GlobalResourceQuota + +The following quota shares CPU and memory across all namespaces belonging to the `green` Tenant: + +```yaml +apiVersion: capsule.clastix.io/v1beta2 +kind: GlobalResourceQuota +metadata: + name: green-shared-compute +spec: + namespaceSelectors: + - matchLabels: + capsule.clastix.io/tenant: green + quota: + hard: + limits.cpu: "8" + limits.memory: 16Gi + requests.cpu: "8" + requests.memory: 16Gi +``` + +Each `GlobalResourceQuota` represents one native `ResourceQuotaSpec`. Create multiple `GlobalResourceQuota` objects when different scopes, selectors, or independently managed limits are required. + +## Namespace selection + +Namespaces are selected through `.spec.namespaceSelectors`. + +Requirements inside one selector are combined with AND. Multiple entries in `namespaceSelectors` are combined with OR. A namespace matching any entry is included only once. + +```yaml +apiVersion: capsule.clastix.io/v1beta2 +kind: GlobalResourceQuota +metadata: + name: production-and-staging +spec: + namespaceSelectors: + - matchLabels: + company.example/environment: production + matchExpressions: + - key: company.example/quota-enabled + operator: In + values: ["true"] + - matchLabels: + company.example/environment: staging + quota: + hard: + requests.cpu: "20" + requests.memory: 40Gi +``` + +Selection behavior: + +- Omitting `namespaceSelectors` selects no namespaces. +- A selector entry containing an empty `matchLabels`/`matchExpressions` selector matches every namespace. +- A selector with a `nil` label selector is ignored. +- Terminating namespaces are not included. +- Namespace membership is recalculated when namespace labels or the quota selectors change. + +Be careful with an empty selector: + +```yaml +spec: + namespaceSelectors: + - {} +``` + +This intentionally shares the quota across all active namespaces in the cluster. + +### Selecting Tenant namespaces + +Capsule labels Tenant namespaces with the Tenant name. The compatibility label can be used directly: + +```yaml +spec: + namespaceSelectors: + - matchLabels: + capsule.clastix.io/tenant: solar +``` + +Multiple Tenants can share one quota: + +```yaml +spec: + namespaceSelectors: + - matchLabels: + capsule.clastix.io/tenant: solar + - matchLabels: + capsule.clastix.io/tenant: wind +``` + +## Quota resources + +`.spec.quota` is a Kubernetes `ResourceQuotaSpec`. The same hard-resource names and scope rules used by Kubernetes apply. + +### Compute resources + +```yaml +spec: + namespaceSelectors: + - matchLabels: + company.example/team: payments + quota: + hard: + requests.cpu: "16" + requests.memory: 32Gi + limits.cpu: "32" + limits.memory: 64Gi + pods: "100" +``` + +Pod requests and limits are calculated using the Kubernetes pod resource helpers. This includes: + +- Regular containers. +- Init containers, including restartable sidecars. +- Pod overhead. +- Pod-level CPU and memory resources when supported by the Kubernetes API server. +- Pod lifecycle behavior, such as excluding terminal Pods from compute usage while retaining `count/pods`. + +Pod-level resources can be placed on a Pod template: + +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: nginx + namespace: solar-production +spec: + replicas: 3 + selector: + matchLabels: + app: nginx + template: + metadata: + labels: + app: nginx + spec: + resources: + requests: + cpu: 100m + memory: 256Mi + limits: + cpu: "1" + memory: 1Gi + containers: + - name: nginx + image: nginx:1.27 +``` + +Kubernetes must support and enable Pod-level resources. If the API server removes the field because the feature is unavailable, quota validation falls back to Kubernetes' historical per-container CPU and memory requirements. + +### Ephemeral storage + +Ephemeral-storage requests and limits are accounted like native Kubernetes Pod quotas: + +```yaml +apiVersion: capsule.clastix.io/v1beta2 +kind: GlobalResourceQuota +metadata: + name: shared-ephemeral-storage +spec: + namespaceSelectors: + - matchLabels: + company.example/storage-budget: shared + quota: + hard: + ephemeral-storage: 100Gi + requests.ephemeral-storage: 100Gi + limits.ephemeral-storage: 200Gi +``` + +Define ephemeral-storage requests and limits on containers: + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: worker + namespace: solar-production +spec: + containers: + - name: worker + image: example.com/worker:latest + resources: + requests: + ephemeral-storage: 2Gi + limits: + ephemeral-storage: 4Gi + volumeMounts: + - name: cache + mountPath: /cache + volumes: + - name: cache + emptyDir: + sizeLimit: 8Gi +``` + +An `emptyDir.sizeLimit` is a volume limit, not a Pod resource request. It does not add `8Gi` to ResourceQuota usage. Configure container `requests.ephemeral-storage` when storage must be reserved and counted. + +### Object counts + +Both legacy core-resource names and generic object-count names are supported. + +```yaml +spec: + quota: + hard: + pods: "100" + services: "20" + secrets: "50" + count/configmaps: "50" + count/deployments.apps: "30" + count/horizontalpodautoscalers.autoscaling: "10" +``` + +Generic count syntax is: + +```text +count/. +``` + +The API group is omitted for core resources: + +```text +count/configmaps +count/pods +``` + +For grouped or custom resources, include the group: + +```text +count/deployments.apps +count/jobs.batch +count/widgets.platform.example.com +``` + +Object creation is reserved through the same atomic ledger as compute resources. For example, if `services: 5` is shared by two namespaces and ten Services are created concurrently, only five admissions can succeed. + +### Persistent storage + +PVC accounting includes the native Kubernetes quota resource names: + +```yaml +spec: + quota: + hard: + persistentvolumeclaims: "20" + requests.storage: 500Gi + fast.storageclass.storage.k8s.io/persistentvolumeclaims: "10" + fast.storageclass.storage.k8s.io/requests.storage: 250Gi +``` + +PVC storage-class resources, allocated storage, resize status, and supported `VolumeAttributesClass` scopes follow Kubernetes ResourceQuota behavior. + +### Scopes and scope selectors + +Native quota scopes can be used without Capsule-specific syntax: + +```yaml +apiVersion: capsule.clastix.io/v1beta2 +kind: GlobalResourceQuota +metadata: + name: high-priority-compute +spec: + namespaceSelectors: + - matchExpressions: + - key: capsule.clastix.io/tenant + operator: Exists + quota: + hard: + requests.cpu: "20" + requests.memory: 40Gi + scopeSelector: + matchExpressions: + - scopeName: PriorityClass + operator: In + values: ["high"] +``` + +Pod scopes such as `Terminating`, `NotTerminating`, `BestEffort`, `NotBestEffort`, `PriorityClass`, and `CrossNamespacePodAffinity` are evaluated before usage is reserved. + +## Atomic admission + +Native `ResourceQuota.status.used` is eventually consistent. Reading the status and then allowing a request is insufficient because multiple requests can observe the same available capacity. + +Global quota admission uses reservations: + +1. The webhook finds every `GlobalResourceQuota` matching the request namespace. +2. The incoming object is decoded and evaluated once. +3. Only resources present in each quota's `hard` list are considered. +4. For updates, only the positive usage delta is reserved. +5. The webhook atomically adds the delta to the quota's `QuantityLedger` using Kubernetes optimistic concurrency. +6. Admission is denied if observed usage plus active reservations would exceed any hard limit. +7. The controller removes reservations as native `ResourceQuota.status.used` catches up. + +This process is applied to every matching `GlobalResourceQuota`. If any matching quota rejects the request, admission is denied and reservations already made for that request are rolled back. + +Dry-run requests are evaluated but do not persist reservations. + +### Readiness and failure behavior + +Admission fails closed while a matching quota is not ready. The ledger must: + +- Refer to the current `GlobalResourceQuota` UID. +- Have observed the current quota generation. +- Contain the request namespace. +- Be initialized from every selected native ResourceQuota. + +The chart enables the webhook with `failurePolicy: Fail` by default: + +```yaml +webhooks: + hooks: + globalresourcequotas: + enabled: true + failurePolicy: Fail +``` + +The webhook must observe namespaced create and update requests across API groups. Narrowing its webhook rules, namespace selector, object selector, or match conditions can create accounting gaps. + +Reservations expire after two minutes if native usage never appears, for example when a later admission plugin rejects the object. A ledger accepts at most 1024 active reservations at one time. + +## Tenant rules + +Tenant rules can generate `GlobalResourceQuota` objects: + +```yaml +apiVersion: capsule.clastix.io/v1beta2 +kind: Tenant +metadata: + name: solar +spec: + owners: + - name: alice + kind: User + rules: + - namespaceSelector: + matchLabels: + company.example/tier: application + quota: + - hard: + requests.cpu: "8" + requests.memory: 16Gi + limits.cpu: "8" + limits.memory: 16Gi + - hard: + services: "20" + count/horizontalpodautoscalers.autoscaling: "10" +``` + +Capsule creates one `GlobalResourceQuota` per entry in the rule's `quota` list. The generated selector combines: + +- The rule's namespace selector. +- The Tenant membership label. + +The Tenant membership requirement prevents a rule from selecting another Tenant's namespaces. Generated quotas are reconciled and pruned with the Tenant rule lifecycle. + +Quota accounting is independent of a rule's request audience. An audience can limit other rule behavior but does not partition the shared resource budget. + +Existing `Tenant.spec.resourceQuotas` behavior remains available for compatibility. Use rule-generated or directly managed `GlobalResourceQuota` objects when an atomic shared limit across namespaces is required. + +## Status + +The status exposes the selected namespaces and observed usage: + +```yaml +status: + observedGeneration: 1 + namespaceCount: 2 + namespaces: + - solar-production + - solar-staging + total: + hard: + requests.cpu: "8" + requests.memory: 16Gi + used: + requests.cpu: "3" + requests.memory: 6Gi + available: + requests.cpu: "5" + requests.memory: 10Gi + namespaceUsage: + solar-production: + used: + requests.cpu: "2" + requests.memory: 4Gi + solar-staging: + used: + requests.cpu: "1" + requests.memory: 2Gi + conditions: + - type: Ready + status: "True" + reason: Succeeded + message: reconciled +``` + +`status.total.used` is observed native usage. Very recent admissions may still exist only as ledger reservations, so admission can correctly reject new work before the observed status or metrics reach the hard limit. + +Useful commands: + +```shell +kubectl get globalresourcequotas +kubectl get globalquota green-shared-compute -o yaml +kubectl get resourcequotas -A \ + -l projectcapsule.dev/global-resource-quota=green-shared-compute +kubectl get quantityledgers -n capsule-system \ + -l projectcapsule.dev/global-resource-quota=green-shared-compute \ + -o yaml +``` + +Managed native ResourceQuota objects and QuantityLedgers are implementation details. Do not edit or delete them directly. + +## Updating a quota + +Hard limits can be increased normally. + +A hard resource cannot be reduced below currently allocated usage. It also cannot be removed while observed usage or active reservations for that resource are non-zero. Release the resources first, wait for status reconciliation, and then reduce or remove the limit. + +When selectors or hard limits change, admission waits for the ledger to observe the new generation and selected namespace set. This prevents requests from being admitted against stale quota state. + +If several `GlobalResourceQuota` objects select the same namespace, all matching quotas apply. A request must satisfy every matching hard limit and scope. + +## Monitoring + +The following Prometheus metrics are exposed: + +| Metric | Labels | Description | +| --- | --- | --- | +| `capsule_global_resource_quota_condition` | `global_resource_quota`, `condition` | Current condition status, where true is `1`. | +| `capsule_global_resource_quota_limit` | `global_resource_quota`, `resource` | Shared hard limit. | +| `capsule_global_resource_quota_usage` | `global_resource_quota`, `resource` | Observed aggregate usage. | +| `capsule_global_resource_quota_available` | `global_resource_quota`, `resource` | Observed available capacity. | +| `capsule_global_resource_quota_usage_percentage` | `global_resource_quota`, `resource` | Aggregate usage as a percentage of the hard limit. | +| `capsule_global_resource_quota_namespace_usage` | `global_resource_quota`, `target_namespace`, `resource` | Observed usage per namespace. | +| `capsule_global_resource_quota_namespace_usage_percentage` | `global_resource_quota`, `target_namespace`, `resource` | Namespace usage as a percentage of the shared limit. | + +Example alerts: + +```yaml +apiVersion: monitoring.coreos.com/v1 +kind: PrometheusRule +metadata: + name: capsule-global-resource-quota-alerts +spec: + groups: + - name: capsule-global-resource-quotas.rules + rules: + - alert: CapsuleGlobalResourceQuotaHighUsage + expr: capsule_global_resource_quota_usage_percentage > 90 + for: 10m + labels: + severity: warning + annotations: + summary: Global resource quota usage is high + description: >- + Resource {{ $labels.resource }} in GlobalResourceQuota + {{ $labels.global_resource_quota }} is at {{ $value }} percent. + + - alert: CapsuleGlobalResourceQuotaNotReady + expr: >- + capsule_global_resource_quota_condition{condition="Ready"} == 0 + for: 10m + labels: + severity: warning + annotations: + summary: Global resource quota is not ready + description: >- + GlobalResourceQuota {{ $labels.global_resource_quota }} + has not been ready for ten minutes. +``` + +## GlobalResourceQuota or ResourcePool? + +Both resources are cluster-scoped and select namespaces, but they solve different allocation problems. + +| | GlobalResourceQuota | ResourcePool | +| --- | --- | --- | +| Main purpose | Enforce one immediate shared ceiling. | Allocate capacity to namespace claims. | +| Consumer object | None. Namespaced objects consume the budget directly. | `ResourcePoolClaim`. | +| Admission behavior | Atomically reserves real object usage. | Schedules and binds claims from pool capacity. | +| Namespace distribution | Dynamic; any selected namespace can consume available capacity. | Explicitly allocated through claims. | +| Queue | No user-visible queue. Admission succeeds or is denied. | Claims can remain queued until capacity is available. | +| Typical use | Tenant-wide compute, storage, or object-count limit. | Delegated self-service allocation from a platform-owned pool. | + +Use `GlobalResourceQuota` when namespaces should compete directly for one shared hard limit. Use a `ResourcePool` when users should explicitly claim and release portions of platform capacity. + +## Troubleshooting + +### `QuantityLedger ... is not initialized` + +The controller is waiting for every selected native ResourceQuota to report its hard and used status. Check: + +```shell +kubectl get globalquota -o yaml +kubectl get resourcequotas -A \ + -l projectcapsule.dev/global-resource-quota= +kubectl get quantityledgers -n capsule-system \ + -l projectcapsule.dev/global-resource-quota= \ + -o yaml +``` + +Also verify that Capsule can list the selected namespaces and reconcile ResourceQuota status. + +### `resource exceeds GlobalResourceQuota` + +The denial contains requested, allocated, and hard resource lists. `allocated` includes both observed usage and active admission reservations, so it can temporarily be higher than `status.total.used`. + +### `must specify requests.cpu ... for: ` + +Kubernetes historically requires every regular and init container to declare CPU or memory resources when those resources are tracked by quota. + +When supported Pod-level resources are present, Capsule follows Kubernetes and skips that legacy per-container requirement. Inspect the Pod or the controller's stored Pod template: + +```shell +kubectl get deployment -n -o yaml +kubectl get replicaset -n -l app=