AI Hardware

CUDA 13.4 GPU Fractions: The Enforcement Gap in Kubernetes

Why a scheduler's GPU-memory reservation is not an enforced limit, how CUDA 13.4 and GPU Fractioning close the gap, and what to test before multi-tenant rollout.

Approximately 9 min read

A Kubernetes scheduler can place two containers on one GPU because their declared memory requests fit. That decision does not, by itself, stop either container from allocating more device memory than promised. This distinction—placement accounting versus runtime enforcement—is the practical problem behind NVIDIA’s October 27, 2026 announcement.

CUDA 13.4 introduces memory-enforced GPU sharing, and the open-source GPU Fractioning Operator connects that capability to Kubernetes. The interesting engineering question is not simply whether a cluster can pack more inference workers onto one accelerator. It is whether an allocation boundary is actually applied before a container starts, remains observable, and fails safely when part of the GPU stack is misconfigured.

This is a source-level architecture analysis and rollout checklist, not a report of tests run on a cluster. All numbers in the examples are illustrative allocations, not measured performance.

The three boundaries people confuse

Scheduling boundary. The scheduler accounts for a workload’s requested share and chooses a GPU with sufficient available capacity. A request is a reservation decision, not necessarily an allocation cap inside the CUDA process. KAI’s older GPU-sharing documentation explicitly warns that its default sharing path does not enforce GPU-memory limits.

Memory boundary. The runtime and driver constrain how much device memory a workload may allocate. CUDA 13.4 adds an application-transparent hard limit and a request/limit model: a job can reserve a guaranteed amount and, when free capacity exists, grow toward a separate maximum. The driver must enforce the ceiling rather than relying on the application’s cooperation.

Fault and compute boundary. How GPU execution is shared—and which jobs can affect one another when something fails—is a different decision. Time-slicing and SM-sharing do not have identical error domains. Neither should be equated with the hardware partitioning offered by MIG.

A platform can implement the first boundary and still fail at the second. It can enforce the second while making an inappropriate assumption about the third. Treat these as three independent acceptance criteria.

The enforcement chain: scheduler, NRI, MPS, driver

The public GPU Fractioning repository identifies four components with separate responsibilities.

KAI Scheduler, or another scheduler that writes the required annotations, chooses a physical GPU and assigns a fractional allocation. The GPU Fractioning controller reconciles a cluster-scoped GpuFractioningConfig and deploys node daemons. Its mpsd component supervises NVIDIA MPS. Its fractiond component is a Node Resource Interface (NRI) plugin that acts during container creation. The metricsd sidecar exposes per-pod usage.

The critical transition happens at container creation. fractiond reads the per-container allocation annotations, injects NVIDIA_GPU_MEMORY_REQUEST and NVIDIA_GPU_MEMORY_LIMIT, supplies the MPS connection and GPU assignment, and lets the NVIDIA driver enforce the limit. The operator documentation says a container that exceeds the enforced limit is terminated with an out-of-memory condition. Application-visible error details can vary; a rollout test should record both the CUDA error and the container outcome.

The sequence matters. A controller that merely annotates a Pod after its GPU process has started cannot be assumed to retrofit the same guarantee. The relevant NRI hook must run, the container toolkit must apply the injected settings, and the installed driver must support the feature.

This is also why installing only a newer CUDA userspace library is not enough. The operator’s published prerequisites include a compatible runtime, GPU Operator operands, and an r615-or-newer NVIDIA driver.

Request is not limit: a concrete allocation

Consider a fictional inference worker requesting 8 GiB but allowed to grow to 12 GiB. The per-container annotations use the actual container name, not a generic GPU resource name:

apiVersion: v1
kind: Pod
metadata:
  name: fractional-inference-example
  annotations:
    nvidia.com/container.worker.gpu-memory.request: "8Gi"
    nvidia.com/container.worker.gpu-memory.limit: "12Gi"
    nvidia.com/container.worker.gpu-compute.mode: "time-slicing"
spec:
  schedulerName: kai-scheduler
  containers:
    - name: worker
      image: nvcr.io/nvidia/cuda:13.0.2-base-ubuntu24.04
      command: ["sleep", "infinity"]

This is a configuration illustration, not a complete tested deployment manifest. The example image tag is taken from the vendor’s published pattern; the node’s driver, operator, runtime and scheduler configuration—not the image tag alone—determine whether memory enforcement is active.

The request of 8 GiB is the scheduling reservation. The 12 GiB limit is the upper allocation ceiling, not 12 GiB of permanently reserved physical capacity. If the workload asks for 10 GiB while neighboring workloads are quiet, that is within its configured ceiling; it does not prove that 10 GiB is available at that instant. A correct scheduler must reason about guaranteed requests, while the node enforces maximum growth.

The operator documentation has a non-obvious default: if only one of request or limit is supplied, the other is populated with the same value. A request-only annotation is therefore not an unlimited burst allowance. This is safer than assuming omitted limits mean infinity, but it is easy to miss when migrating from older GPU-sharing schemes.

KAI’s October 2026 NvFractions mode also supports proportion-based requests and limits. Do not silently combine the old fraction-based annotations with absolute memory values without checking the current admission rules. A fractional request across multiple GPUs is per device, not one aggregate pool: a 40 GiB request on four devices means 40 GiB on each device, as NVIDIA’s announcement explains.

Compute mode changes the failure model

The default time-slicing mode lets workloads take turns using the GPU. NVIDIA describes separate error domains for those workloads. Its SM-sharing mode uses MPS to let workloads occupy streaming multiprocessors concurrently; those workloads share an error domain. In other words, selecting SM-sharing for higher concurrency is also an isolation decision.

Neither mode is a promise of a fixed fraction of peak FLOPS. NVIDIA’s approximate 1/n description applies when n workloads are fully busy, while idle capacity can be redistributed. Actual latency depends on kernel shape, memory pressure, synchronization, context behavior and load. A model-serving team should measure tail latency and interference rather than infer service-level guarantees from the memory fraction.

The scheduler must also avoid placing time-slicing and SM-sharing workloads on the same GPU simultaneously. The NVIDIA design description states that KAI tracks this per GPU, so different GPUs in one node can use different modes.

MIG remains a different tool: it partitions supported hardware into isolated GPU instances. NVIDIA’s GPU Operator time-slicing guide explicitly distinguishes time-slicing from MIG’s memory and fault isolation. A memory allocation cap is not a substitute for evaluating tenant trust, GPU side channels, driver attack surface or hardware fault domains.

The deployment prerequisites are a release gate, not a footnote

The GPU Fractioning README currently specifies Kubernetes 1.28 or later; containerd 2.0 or later with NRI enabled, or CRI-O with NRI support; an r615-or-newer NVIDIA driver; and appropriate NVIDIA GPU Operator components. The operator’s minimum toolkit is v1.20.1 because it supplies the apply-cuda-memory-limits CDI hook; the device-plugin minimum is v0.20.1.

The README recommends GPU Operator v26.7.1 because its bundled operands meet those minimums, but it warns that the required r615 driver is not selected by default. An older GPU Operator can sometimes work with explicit compatible operand overrides; the operator version alone is not a complete compatibility check.

One particularly dangerous shortcut is skipGpuStackVersionChecks. The project’s own documentation warns that bypassing version checks can make the operator report Ready even when the underlying GPU stack cannot enforce limits. That would produce the exact failure this architecture is meant to prevent: a scheduler accounts for fractions while memory remains effectively unfenced.

A credible rollout gate should therefore verify the effective toolkit, device-plugin and driver versions, not merely a successful Helm command or a green controller status. If versions are pinned by opaque image digests, the README notes that the operator may not be able to infer their compatibility. Independent inspection becomes essential.

There is a second operational trap around NRI. If the GPU Operator runs its own NRI plugin, the default NVIDIA RuntimeClass path changes; the GPU Fractioning README documents additional namespace allowlist requirements. A Pod that starts without the expected device injection is not evidence that sharing works. Confirm the actual GPU assignment and enforced cap.

Observability: measure the fraction, not the whole GPU

GPU utilization from a node-wide tool can be misleading when multiple Pods share one accelerator. The operator exposes per-pod metrics including gpu_fractioning_gpu_memory_used_bytes, gpu_fractioning_gpu_sm_utilization_percent and gpu_fractioning_gpu_sm_utilization_percent_normalized. The normalized SM metric is divided by the Pod’s assigned fraction and capped at 100%; it is not a new measure of physical GPU capacity.

The controller also publishes ready, desired and degraded node counts. Node condition gpu-fractioning.nvidia.com/Ready and GpuFractioningConfig conditions provide a more useful readiness view than simply checking that a scheduler Pod is running.

The operator’s observability section says its Prometheus scrape endpoints use plain HTTP. That is acceptable only within an appropriately protected network boundary; operators should restrict scrape access rather than assuming metrics endpoints are authenticated by default.

Useful dashboard questions are operational: which Pod owns each memory request and limit, which GPU is assigned, what is its peak observed allocation, did a container hit its ceiling, and did the node lose enforcement during a driver upgrade? Aggregate GPU utilization alone cannot answer them.

A falsifiable rollout test, including failure cases

The following is a proposed validation plan for a disposable test cluster. No test results are claimed.

  1. Version gate: record node driver, toolkit, device plugin, container runtime and NRI configuration. Verify the expected Ready conditions before placing fractional workloads.
  2. Allocation ceiling: run a CUDA allocation probe inside a Pod with an 8 GiB request and 12 GiB limit. Increase allocations in controlled steps and record where they fail. Confirm the limit is enforced at the configured ceiling, allowing for allocator overhead and device-reserved memory.
  3. Neighbor protection: co-locate a second small GPU workload. Have the first deliberately exceed its cap; verify that the neighbor continues operating and its memory allocation is not displaced by the first Pod’s excess request.
  4. Burst semantics: start one workload above its request but below its limit while the GPU is otherwise idle, then add a neighbor. Observe admission, allocation outcomes and reclamation behavior; do not assume that a request/limit range implies automatic live preemption of GPU allocations.
  5. Compute modes: compare latency distributions under time-slicing and SM-sharing using identical models, pinned software, equal workload mixes and identical memory allocations. Record both median and p95/p99 latency, plus errors.
  6. Fail-closed behavior: deliberately disable or misconfigure the NRI hook or use an incompatible stack in an isolated environment. The test passes only if the platform detects the missing enforcement and refuses to present the GPU as safely fractioned.
  7. Upgrade lifecycle: observe the node’s enforcement readiness as the NVIDIA driver is drained and upgraded. The operator documents coordination through GPU Operator node labels and daemon draining; verify the sequence on the exact versions deployed.

The most important negative control is step 6. Two Pods sharing a GPU successfully is weak evidence: traditional time-slicing can already do that. The feature is proven only when an over-budget allocation is constrained and missing enforcement is detectable.

What this changes—and what it does not

CUDA 13.4 and GPU Fractioning close a specific gap between Kubernetes GPU placement and driver-level memory enforcement. That can make fractional allocation more predictable for small inference servers, notebooks and batch jobs. The source material does not establish a universal utilization gain, a cost-saving percentage or a benchmark speedup. Those claims require a workload-matched experiment.

The architecture is useful precisely because its boundaries are explicit: the scheduler chooses capacity, NRI and the container toolkit apply the settings, the driver enforces the memory ceiling, and MPS determines compute-sharing behavior. Treat each as a separate testable link.

For production decisions, the acceptance criteria should be enforced memory caps, verified failure behavior, known error domains, observable per-Pod usage, and upgrade-safe readiness—not just a higher count of Pods per GPU.

Primary sources

Sources and further reading