PyTorch CUDA OOM: Allocated vs Reserved GPU Memory

A PyTorch workload can fail with torch.OutOfMemoryError: CUDA out of memory even when the GPU seems to have enough memory based on the model’s weight size, or even when two tools report apparently contradictory memory totals.

The first question is not “Which environment variable should I copy?” It is which allocator owns which bytes, and which allocation actually failed?

PyTorch’s CUDA memory-management documentation distinguishes allocated memory (typically active tensors) from reserved memory (memory held by PyTorch’s caching allocator). They are not interchangeable and neither alone is a complete picture of total device usage.

Three different memory views

For example, an invented diagnostic snapshot might show 11 GB allocated, 15 GB reserved, and 18 GB of total GPU memory reported as in use. The gap between 11 and 15 GB is not automatically a leak. Some reserved bytes may be reusable cached blocks. The remaining discrepancy can include CUDA context and allocations from libraries outside PyTorch.

Those numbers are illustrative only. The actual interpretation depends on the device, allocator backend, process, and when the measurements were collected.

Start with an in-process diagnostic

Run this in the same Python process that experiences the OOM, preferably just before its failing operation:

import torch

if not torch.cuda.is_available():
    raise RuntimeError("CUDA is unavailable in this process")

device = torch.cuda.current_device()
gib = 1024 ** 3

print("GPU:", torch.cuda.get_device_name(device))
print("allocated GiB:", round(torch.cuda.memory_allocated(device) / gib, 3))
print("reserved GiB:", round(torch.cuda.memory_reserved(device) / gib, 3))
print("peak allocated GiB:", round(torch.cuda.max_memory_allocated(device) / gib, 3))
print(torch.cuda.memory_summary(device=device, abbreviated=True))

These APIs show data available to that process’s allocator. Running the snippet in a brand-new Python interpreter does not reconstruct the allocations held by a different server process. If the OOM prevents the service from continuing, add bounded instrumentation to a controlled reproduction and preserve the failing exception.

The official memory_summary reference describes the summary as useful for periodic observation and OOM diagnosis.

Is cached memory the reason you ran out?

Not necessarily.

PyTorch’s caching allocator avoids returning device memory on every tensor deletion, because constant allocation and free operations can be expensive and introduce synchronization overhead. The allocator may retain unused blocks for later allocations. The CUDA semantics documentation explicitly explains why nvidia-smi can show that memory as used.

Calling:

torch.cuda.empty_cache()

returns unused cached segments where the allocator can release them. It does not release memory still referenced by live tensors and does not magically increase the amount of memory PyTorch can use while those tensors remain necessary.

If the live working set itself exceeds what the device can accommodate, reducing cache does not solve the peak-memory problem.

Diagnose four common causes separately

1. The live working set is genuinely too large

Large activation tensors, attention intermediates, batch expansion, larger resolution, optimizer states, or unexpectedly retained tensors can grow the active set. For inference, model weights are only one component: KV caches, temporary buffers and batch characteristics also matter.

Test a controlled reduction in batch size or sequence length, keeping all other relevant settings fixed. If peak allocated memory falls and the workload succeeds, that supports a genuine capacity explanation. It does not prove fragmentation was the cause.

2. Old tensors remain referenced

A Python list that retains outputs from every iteration can extend tensor lifetimes. Gradients and computation graphs can likewise be retained accidentally when code stores tensors instead of scalar summaries.

Inspect references and autograd behaviour. For inference, verify torch.inference_mode() or torch.no_grad() is applied where appropriate. Do not indiscriminately detach training outputs if their gradients are required for the task.

3. Fragmentation or allocator policy is involved

Sometimes the total amount of unused memory is not equivalent to one allocatable block with the properties a request requires. Allocator fragmentation and block splitting can matter, but a large reserved - allocated value alone does not prove fragmentation caused this OOM.

Look at the relevant stats or a snapshot and examine the failing allocation. PyTorch documents advanced tuning with PYTORCH_ALLOC_CONF; the older name PYTORCH_CUDA_ALLOC_CONF is retained as a backward-compatible alias in the current manual. Some tuning settings apply only to a particular backend.

Blindly setting max_split_size_mb or toggling the allocator for every OOM can create different allocation behaviour without addressing a live-memory peak. Treat such changes as controlled experiments with pre/post measurements.

4. The memory is outside the PyTorch allocator

The PyTorch memory-snapshot guide warns that it only sees memory managed by the PyTorch allocator. CUDA memory reserved through other APIs or libraries can lie outside its view.

Check actual GPU processes, library-specific allocations, and multi-process service configuration. If nvidia-smi shows substantial usage while PyTorch’s allocator snapshot accounts for little of it, investigate context and non-PyTorch owners before treating it as a Python tensor leak.

Use a memory snapshot when the summary is insufficient

For difficult allocation patterns, PyTorch provides tools to record allocator history and inspect memory snapshots. The Understanding CUDA Memory Usage guide explains the history recorder and visualizer.

A snapshot is a diagnostic artifact. It may capture allocation traces, stack locations, and workload structure. Review it for confidential paths or operational details before sharing. The memory visualizer cannot reveal bytes allocated by components outside the tracked allocator, so a blank region in its accounting does not prove that the device is empty.

What to report for reproducibility

Record:

1. GPU model and total VRAM.
2. PyTorch version and the active CUDA allocator backend.
3. Model precision, batch size, input length and inference/training mode.
4. Allocated, reserved and peak allocated bytes from the failing process.
5. Driver-visible GPU/process usage at the same time.
6. Requested failed allocation size and traceback.
7. One controlled intervention and the observed result, if actually tested.

A useful reproduction does not need a large application. If you can reproduce the same class of failure with a bounded smaller example that preserves the allocation pattern, that is often safer for diagnosis.

A decision tree worth keeping

• If allocated memory is close to practical capacity, reduce the live working set.
• If a cached-memory gap is large, inspect allocator stats and block reuse before labeling it fragmentation.
• If the driver reports usage that PyTorch cannot account for, inspect other CUDA allocations and processes.
• If empty_cache() appears to help once, do not assume it fixes the root cause; compare repeated controlled runs.
• If the failure occurs only after repeated requests, audit tensor lifetime and workload variation.

The central diagnostic rule is: allocated is not reserved, and reserved is not total GPU usage. Measure the distinction before choosing the fix.

Primary references

• PyTorch CUDA semantics and memory management
• Understanding CUDA Memory Usage
• torch.cuda.memory_summary
• torch.cuda.memory_allocated
• torch.cuda.memory_reserved