GPU isolation via a namespace-scoped mutating webhook and composed DRA objects

[devdattakulkarni]

Problem

KubePlus has no notion of GPU isolation today. GPU access is a real multi-tenancy concern for any Kind instance with a GPU-backed component anywhere in its execution graph — not only when the agent's primary model is served locally. Even an agent whose main reasoning loop calls an external LLM API can depend on GPU-backed components elsewhere: an embedding or reranking model behind a RAG tool, a local classifier used for guardrails or PII detection, a vision/OCR/speech tool, or a small local model used for cheap intermediate steps before escalating to an external API. Whether the top-level LLM call goes to an external provider is largely orthogonal to whether GPU isolation matters — what matters is whether any component the instance depends on, directly or via a shared MCP server, touches a GPU.

Kubernetes' GPU story has shifted meaningfully in the last few months: Dynamic Resource Allocation (DRA) graduated to GA in Kubernetes 1.34 and is on by default from there, replacing the old opaque-integer device-plugin model with structured objects (DeviceClass, ResourceClaim, ResourceClaimTemplate, ResourceSlice). NVIDIA has also donated its DRA driver for GPUs to the CNCF, moving it toward community ownership rather than a single-vendor plugin.

Referencing a DRA claim from a Pod requires two things in the Pod spec: a pod-level spec.resourceClaims entry naming the claim source, and a per-container resources.claims entry referencing that name. Requiring a third-party chart's values.yaml to expose fields for either of these is the same unrealistic assumption flagged in the node-isolation issue — a chart not written with KubePlus in mind has no reason to expose a resourceClaims field, and there's no equivalent of a StorageClass-style cluster default to fall back on.

Three GPU sharing strengths exist, with different isolation/utilization trade-offs:

Mechanism	Isolation	Notes
Full/exclusive GPU	Strongest (whole device)	Lowest utilization for bursty agent workloads
MIG (Multi-Instance GPU)	Hardware-partitioned, up to 7 instances on A100/H100/H200/B200	Fixed-size partitions; supported on Ampere-generation+ GPUs only
Time-slicing	None — no memory/fault isolation	High density, unsafe for mutually untrusting tenants
MPS	Soft memory limits only	Better utilization than time-slicing, still not hard isolation

When KubePlus processes this annotation, it writes the resolved claim-template name onto the Namespace object:

apiVersion: v1
kind: Namespace
metadata:
  name: team-a
  labels:
    kubeplus.io/gpu-isolation-enabled: "true"
  annotations:
    kubeplus.io/gpu-isolation-resolved: |
      {"claimTemplateName": "kubeplus-gpu-team-a-agent-instance"}

As with node isolation, the label exists purely so namespaceSelector can match on it cheaply; the annotation carries the only value the webhook actually needs, so the webhook never has to read the instance CR or the ResourceClaimTemplate object at request time — only the Namespace, via its own informer cache.

Mutating webhook configuration changes

A separate webhooks[] entry, independent of both the existing CRD-focused rule and the node-isolation Pod rule (multiple mutating webhook entries on Pods compose fine, since each only patches the specific paths it owns):

{
  "name": "gpu-isolation.kubeplus.io",
  "rules": [
    {
      "apiGroups": [""],
      "apiVersions": ["v1"],
      "operations": ["CREATE"],
      "resources": ["pods"],
      "scope": "Namespaced"
    }
  ],
  "namespaceSelector": {
    "matchExpressions": [
      {
        "key": "kubeplus.io/gpu-isolation-enabled",
        "operator": "In",
        "values": ["true"]
      }
    ]
  },
  "failurePolicy": "Ignore",
  "sideEffects": "None",
  "admissionReviewVersions": ["v1"]
}

failurePolicy: Ignore for the same reason as node isolation: brief webhook unavailability should fail open rather than blocking all pod creation in the cluster. Unlike node isolation, though, failing open here means a pod could start without its GPU claim at all — worth flagging as a real trade-off rather than a purely cosmetic one, since "pod runs but can't see a GPU" is a more disruptive failure mode than "pod runs on any node." If that trade-off isn't acceptable, failurePolicy: Fail is the alternative, at the cost of blocking pod admission in GPU-isolated namespaces during webhook downtime.

JSON patch implementation (Go)

package gpuisolation
import (

"encoding/json"

"fmt"
corev1 "k8s.io/api/core/v1"

)
// GPUIsolationSpec is the resolved value copied onto a Namespace's

// kubeplus.io/gpu-isolation-resolved annotation.

type GPUIsolationSpec struct {

ClaimTemplateName string json:"claimTemplateName"

}
const claimRefName = "kubeplus-gpu"
// namespaceCache is populated by the same Namespace informer used for node

// isolation and kept current via Add/Update/Delete event handlers. Lookups

// are in-memory map reads keyed by namespace name — no API calls happen per

// admission request.

type namespaceCache interface {

Get(namespace string) (*GPUIsolationSpec, bool)

}
// BuildGPUIsolationPatch returns a JSON Patch (RFC 6902) that adds a

// pod-level resourceClaims entry pointing at the instance's

// ResourceClaimTemplate, and references that claim from every container's

// resources.claims list. Existing pod-level resourceClaims and per-container

// resources.claims entries, if any, are preserved rather than clobbered.

func BuildGPUIsolationPatch(pod *corev1.Pod, iso *GPUIsolationSpec) ([]byte, error) {

var patches []map[string]interface{}
claimEntry := map[string]interface{}{
	"name": claimRefName,
	"source": map[string]interface{}{
		"resourceClaimTemplateName": iso.ClaimTemplateName,
	},
}

if len(pod.Spec.ResourceClaims) == 0 {
	patches = append(patches, map[string]interface{}{
		"op":    "add",
		"path":  "/spec/resourceClaims",
		"value": []interface{}{claimEntry},
	})
} else {
	patches = append(patches, map[string]interface{}{
		"op":    "add",
		"path":  "/spec/resourceClaims/-",
		"value": claimEntry,
	})
}

claimRef := map[string]interface{}{"name": claimRefName}

for i, c := range pod.Spec.Containers {
	if len(c.Resources.Claims) == 0 {
		patches = append(patches, map[string]interface{}{
			"op":    "add",
			"path":  fmt.Sprintf("/spec/containers/%d/resources/claims", i),
			"value": []interface{}{claimRef},
		})
	} else {
		patches = append(patches, map[string]interface{}{
			"op":    "add",
			"path":  fmt.Sprintf("/spec/containers/%d/resources/claims/-", i),
			"value": claimRef,
		})
	}
}

return json.Marshal(patches)

}
// HandleAdmission is the webhook's per-request entry point. Given the

// existing namespaceSelector-based filtering at the MutatingWebhookConfiguration

// level, every request reaching here is already known to be a Pod CREATE in a

// namespace labeled kubeplus.io/gpu-isolation-enabled=true.

func HandleAdmission(namespace string, pod *corev1.Pod, cache namespaceCache) (patch []byte, err error) {

iso, ok := cache.Get(namespace)

if !ok {

// Label present but annotation missing/unparseable — fail open,

// consistent with failurePolicy: Ignore at the config level.

return nil, nil

}

return BuildGPUIsolationPatch(pod, iso)

}

Known limitation, called out deliberately rather than hidden: the patch above adds the claim reference to every container in the pod, including sidecars that may not actually touch the GPU. This is a conscious simplification — identifying which specific container needs the device would require chart-specific knowledge (a container name convention, an annotation, etc.) that reintroduces exactly the kind of per-chart assumption this design is trying to avoid. Referencing an unused claim from a sidecar is inert (it grants visibility, not automatic usage), so this is a reasonable default; revisit only if a concrete chart surfaces a problem with it.

Enforcing maxDevices via ResourceQuota

Kubernetes supports quota scoped to DRA claim counts natively: count/resourceclaims.resource.k8s.io. KubePlus adds this key to the instance's existing per-instance ResourceQuota object (the same object storage isolation already extends), set from maxDevices — no admission-policy or webhook logic needed for this part, since ResourceQuota enforcement is built into the API server.

Acceptance criteria

• An instance with no kubeplus.io/gpu-isolation annotation gets no GPU device claim and no Pod mutation (default-deny).
• sharing: mig produces a ResourceClaimTemplate/DeviceClass pair; two instances requesting the same migProfile on the same node get isolated, non-overlapping MIG instances.
• A Pod created in a GPU-isolated namespace receives the pod-level resourceClaims entry and per-container resources.claims reference without the chart itself having declared either field, and without any pre-existing resourceClaims/ resources.claims entries the chart did declare being removed.
• Webhook latency for Pod admission in a GPU-isolated namespace does not meaningfully increase with chart complexity, for the same reason as node isolation: no ownership traversal is performed.
• Exceeding maxDevices is rejected by ResourceQuota, not the webhook or an admission policy.
• Deleting the instance removes the ResourceClaimTemplate, the quota keys, and the namespace label/annotation naturally via namespace deletion, with no explicit cleanup step required.

Demo steps

1. On a node with MIG enabled (or simulated via a MIG-capable GPU in the cluster), instantiate an Agent Kind instance with:

   metadata:  annotations:    kubeplus.io/gpu-isolation: |      {"sharing": "mig", "migProfile": "1g.10gb", "maxDevices": 1}
   

2. Confirm the namespace carries the label and resolved annotation, and that the ResourceClaimTemplate exists:

   kubectl get namespace team-a -o jsonpath='{.metadata.labels}{"\n"}{.metadata.annotations}'kubectl get resourceclaimtemplate -n team-a
   

3. Confirm a pod in that namespace picks up the claim without the chart declaring it:

   kubectl get pod <pod-name> -n team-a -o jsonpath='{.spec.resourceClaims}'kubectl get pod <pod-name> -n team-a -o jsonpath='{.spec.containers[*].resources.claims}'
   

4. Attempt to create a second GPU-requesting pod in the same namespace beyond maxDevices: 1 and show the ResourceQuota rejection.
5. Delete the instance and confirm the ResourceClaimTemplate and quota keys are gone:

   kubectl delete agent team-a-agent-instance -n team-akubectl get resourceclaimtemplate -n team-a   # expect: not found
   

Research note: hybrid MIG + time-slicing for agent workloads

hybrid mode is deliberately included as an open research question rather than a fully specified mechanism: use MIG to give each tenant a hardware-isolated partition as a baseline guarantee, then apply time-slicing within that partition across that same tenant's own bursty invocations to improve utilization, without extending time-slicing's lack of isolation across tenant boundaries. Whether this is worth the added scheduling complexity for agent-specific traffic patterns versus just provisioning a slightly larger MIG partition per tenant is an open evaluation question. Treat hybrid as experimental until there's utilization data comparing it against mig-only and exclusive under realistic agent traffic.