AI-narrated article. Prose is preserved with pronunciation adjustments; code and image cues refer to the written article.
Kubernetes DRA for GPUs: Allocation Is Only Half the Contract
This is the AI-narrated article. The prose follows the written post; pronunciation is adjusted, and code and images are announced separately.
Use explicit GPU claims to distinguish inventory, scheduler allocation, node preparation and runtime injection—and diagnose the boundary that actually failed.
TL;DR: Dynamic Resource Allocation gives GPU workloads an explicit device contract, but an allocated claim does not prove a container can use its GPU. Separate advertised inventory, scheduling, node preparation and runtime injection when designing the platform and diagnosing failures. This article provides a pinned NVIDIA DRA evaluation example that selects one full GPU by memory capacity, explains claim lifecycle and permissions, and distinguishes an unsatisfiable selector from a preparation failure. It also preserves the driver support caveat: stable Kubernetes APIs do not imply production support for every vendor allocation mode.
A GPU allocation is not a running GPU workload
An inference Pod has a node assignment and its ResourceClaim names a GPU, yet the container never starts. Adding replicas cannot repair that boundary. The scheduler has selected a device; the node still has to prepare it and the runtime has to expose it correctly.
Dynamic Resource Allocation makes these responsibilities explicit. DeviceClass defines selection policy, ResourceSlice advertises inventory, and ResourceClaim records an allocation. Kubelet then calls the node-local DRA driver before starting containers. NVIDIA's driver architecture uses CDI, the Container Device Interface, to make the allocated GPU available through the runtime.
This is valuable when an integer such as nvidia.com/gpu: 1 cannot express the device contract: memory capacity, product attributes, sharing policy or multiple related devices. It also introduces operational state that deserves its own ownership and diagnosis.
Choose a baseline instead of enabling every DRA feature
The core DRA API became stable in Kubernetes 1.34. Its stability does not imply that every sharing, health or capacity extension is equally mature. Kubernetes 1.36's DRA update discusses additional features with separate maturity levels.
Use Kubernetes 1.36 and the NVIDIA DRA driver 0.5.0 release as this example's baseline. Request one exclusive full GPU with more than 40 GiB of reported memory. This avoids turning an allocation example into an unverified claim about MIG isolation, MPS performance or multi-node NVLink readiness.
This is an evaluation example. The pinned driver's README says its GPU-allocation features are not officially supported; ComputeDomains have a different support status. Kubernetes API stability does not turn this driver mode into a supported production GPU allocator.
Treat the driver and device-plugin transition as infrastructure work. The driver's prerequisites require working host NVIDIA drivers, NVIDIA Container Toolkit and a CDI-capable runtime. Its GPU allocation mode must not compete with the conventional NVIDIA GPU device plugin on the same devices. A pool advertising the same hardware through two allocators is not additional capacity.
Keep the allocation and preparation evidence separate
Image. DRA sequence showing inventory, allocation, node preparation and runtime handoff
The diagram shows distinct evidence boundaries. Inventory publication proves that devices are advertised. A claim's allocation proves that scheduling selected a device. Neither proves that node preparation or runtime injection succeeded.
Platform owners should monitor these separately: absent or stale inventory, time spent waiting for allocation, repeated preparation failures, and application-level GPU visibility. Record claim UID, node and allocated device identity together when investigating; a Pod name alone can refer to successive incarnations.
A ResourceClaimTemplate creates a separate claim for each consuming Pod. A directly referenced ResourceClaim has a different lifecycle and can be shared deliberately where supported. Choose that lifecycle explicitly; accidental retention of a claim can retain a scarce allocation after the application team believes a workload is finished.
A full-GPU example with a real hardware prerequisite
Run this only in a dedicated GPU test cluster. Use Kubernetes 1.36 with the resource.k8s.io/v1 API, Helm 3.8 or later, matching kubectl and Bash. The driver prerequisites require a directly installed NVIDIA host driver 565 or later, NVIDIA Container Toolkit 1.18.0 or later, a runtime with CDI enabled, and Node Feature Discovery 0.18.2 or later publishing NVIDIA PCI labels. At least one available NVIDIA Data Center GPU must advertise more than 40 GiB of memory. Host drivers must be installed directly on the node for the default driver root used here; this Helm chart does not install them.
Install the pinned chart after confirming the conventional GPU device plugin is absent from this test pool. Chart values expose an explicit GPU-allocation override. Disable ComputeDomain support because this example does not use IMEX channels.
Code example. Refer to the written article for the complete code.
Proceed only when the GPU plugin is ready and ResourceSlices advertise suitable devices. Inspect the actual gpu.nvidia.com memory capacity rather than translating a product marketing label into an assumed schedulable quantity. The full-GPU guide documents this CEL selector and the Pod-to-template references.
Save the following as gpu-check.yaml. The memory expression is a selection predicate; it selects a full device with sufficient capacity. It does not reserve a 40 GiB slice or cap the application's GPU memory consumption.
Code example. Refer to the written article for the complete code.
The container's claim name links to the Pod's claim entry, which links to the template. CPU and host-memory requests remain separate from GPU allocation. No nvidia.com/gpu integer limit is added to this DRA workload.
Code example. Refer to the written article for the complete code.
A successful run should show an allocated device in the generated claim and a visible GPU inside the container. Compare device identity with the driver-published inventory. These are verification instructions and expected behavior, not results from an executed GPU cluster. Listing a GPU is also not a CUDA correctness or throughput benchmark.
Diagnose the boundary that failed
An illustrative failure is a cluster containing only full GPUs with 40 GiB of reported capacity. The selector uses greater than 40 GiB, so none qualify. Inspect inventory, claim allocation and scheduling events before blaming CDI. If the application genuinely supports smaller devices, create a revised template and recreate this disposable test Pod; existing generated claims do not inherit later template edits. Do not relax the requirement for a workload that would then fail with GPU out-of-memory errors.
The allocated-but-not-starting case needs a different investigation. Inspect Pod events, the GPU plugin on the assigned node, kubelet preparation errors and runtime CDI configuration. Check the configured host driver root: a GPU Operator-managed driver directory differs from a directly installed host driver. Repair that integration under the platform's maintenance procedure, then let preparation retry. Deleting every claim would erase evidence and disturb healthy workloads.
Code example. Refer to the written article for the complete code.
Do not set spec.nodeName to force progress. That bypasses the scheduler and its allocation work; Kubernetes documents the resulting pre-scheduled Pod failure. Use a node selector when placement must be constrained while preserving normal scheduling.
Make GPU policy an owned platform contract
Let platform administrators own DeviceClasses, driver configuration and approved sharing modes. Give workload teams namespaced claim/template permissions with admission controls for permitted classes and configurations. A CEL predicate selects inventory; it is not authorization or a substitute for hardware isolation.
Keep ordinary workload namespaces outside DRA admin-access policy. Kubernetes RBAC good practices describes the privileged namespace label resource.kubernetes.io/admin-access: "true", which permits claims accessing devices allocated across namespaces. That is a maintenance capability, not a shortcut for a Pending application. Control permission to modify this namespace label as carefully as claim creation.
For this exclusive test cluster, clean up the workload first and inspect remaining claims before removing the driver:
Code example. Refer to the written article for the complete code.
For production adoption, require evidence at all four boundaries: advertised inventory, intended allocation, successful node preparation and application GPU use. DRA improves the contract; the platform still owns its operational completion.
Frequently asked questions
Does a GPU memory selector reserve only that amount of VRAM? No. This example filters full devices by published capacity; it does not partition or cap their memory.
Does an allocated ResourceClaim prove the container can use its GPU? No. Node preparation, CDI runtime integration and application verification remain separate steps.
Can I keep the conventional device plugin enabled for the same test GPUs? Not for this pinned driver configuration. Resolve allocator ownership before enabling full-GPU DRA allocation.