CUDA Graph Replay: Static Addresses and the Real CPU Launch Trade-off

A GPU can finish a small kernel quickly and still spend much of an inference iteration waiting for the CPU to issue the next one. CUDA Graphs address this submission overhead by recording a compatible sequence of GPU operations and replaying it with much less repeated host-side setup. They do not make every model inherently faster, remove memory constraints, or turn arbitrary dynamic Python into a reusable graph.

The important engineering question is not merely whether graph replay works. It is which part of the workload can be captured safely, how often that captured work can be reused, and whether the saved CPU launch time exceeds capture, memory, and shape-management costs. This guide draws from the CUDA Programming Guide, PyTorch CUDA semantics, and NVIDIA’s PyTorch graph practices. No latency measurements in this article are presented as RAMGPT experiments.

What a CUDA graph actually captures

Ordinary eager execution submits GPU work incrementally. Python invokes framework code; the runtime configures a kernel launch; the GPU executes it; then the host submits more work. For a few large, long-running kernels, host launch overhead may be a small part of the total. For many short kernels, the gaps can become material.

A CUDA graph represents operations as nodes and dependencies as edges. CUDA separates graph definition, instantiation, and repeated execution. An instantiated graph can be launched many times without reconstructing the same submission sequence on every iteration. The official programming guide describes kernel, memory-copy, event, and other graph node types; a graph is not limited to a single kernel.

This is why graph replay can improve the host-to-device submission path. It is not a blanket claim that GPU arithmetic has become cheaper. NVIDIA’s straight-line graph analysis reports launch-overhead behavior for specific CUDA versions, graph topologies, and GPUs; those figures should not be copied into an LLM benchmark without matching its conditions.

The three different costs to measure

An average request latency hides these stages if it combines cold starts and warm replays. Conversely, reporting only the replay kernel time hides the cost of moving new inputs into the graph’s persistent buffers and returning useful results.

For an inference service, record both end-to-end request latency and GPU execution time, plus the distribution of batch sizes and sequence lengths. Graph capture may improve a low-concurrency, launch-bound decode path without meaningfully improving a long, compute-bound prefill path. That is a hypothesis to profile, not a universal property of every serving engine.

Why memory addresses are part of the contract

PyTorch’s graph replay runs captured kernels with the same pointer arguments used during capture. The contents of those buffers may change; the addresses must remain valid. If a program replaces a tensor with a fresh allocation and assumes the graph will follow the new Python variable, it can replay against the original address instead.

The safe mental model is:

capture: kernels refer to persistent input/output storage
replay 1: write new values into persistent input storage -> replay
replay 2: write different values into that same storage -> replay

That is different from:

capture: kernels refer to tensor A
replay: assign a new tensor B to the Python name A
expectation: graph automatically follows B

The second expectation is incorrect. A Python name change does not rewrite the captured kernel arguments. Keep graph input and output tensors alive; update input in place, for example with a copy into preallocated storage.

PyTorch also uses graph-private allocator pools to protect addresses required by replay. These pools can raise retained GPU memory, especially when many graphs are captured. Pool sharing is an advanced optimization with replay-order and concurrency constraints; blindly sharing memory between independently replayed graphs risks incorrect behavior.

A minimal PyTorch replay illustration

The following example demonstrates the storage contract with a fixed-shape vector. It requires a working CUDA-enabled PyTorch installation and a compatible NVIDIA GPU. It is a teaching example, not a performance benchmark.

import torch

if not torch.cuda.is_available():
    raise RuntimeError("CUDA is required")

static_input = torch.zeros(4096, device="cuda")
warmup_stream = torch.cuda.Stream()
warmup_stream.wait_stream(torch.cuda.current_stream())

with torch.cuda.stream(warmup_stream):
    for _ in range(3):
        warmup_output = static_input.square() + 1

torch.cuda.current_stream().wait_stream(warmup_stream)

graph = torch.cuda.CUDAGraph()
with torch.cuda.graph(graph):
    static_output = static_input.square() + 1

for value in (2.0, 3.0):
    static_input.fill_(value)
    graph.replay()
    torch.cuda.synchronize()
    expected = static_input.square() + 1
    torch.testing.assert_close(static_output, expected)
    print(value, static_output[0].item())

The two printed results should be 5 and 10. Notice that the graph is captured once; later iterations mutate the same input storage and replay the captured work. The synchronization and scalar read are intentionally outside capture. The example keeps output storage referenced and compares each replay against an eager calculation.

A graph-safe implementation should perform the relevant warmup on a side stream before capture, as documented by PyTorch. Real models may need additional setup, especially for distributed operations, custom kernels, mixed precision, and RNG. Do not assume that success on a vector operation proves that an entire transformer forward pass is graph-safe.

Why seemingly harmless Python breaks capture semantics

Graph replay does not rerun arbitrary Python statements from the capture block. CPU-side tokenization, file reads, Python list updates, ordinary print calls, and host-side conditionals are not magically re-executed by replay. PyTorch also prohibits CPU/GPU synchronization operations such as extracting a GPU scalar with a synchronous scalar read during capture.

Data-dependent branches are a separate issue. A graph recorded for one control-flow path is not a dynamic interpreter for all future paths. Capture the stable GPU region and keep host orchestration outside it, or use a framework-supported mechanism designed for the dynamic operation. The CUDA programming guide supports richer graph node types, but those lower-level features do not make arbitrary Python branches graph-safe.

Similarly, a captured graph assumes stable tensor shapes and layouts for its captured operations. If an inference engine serves varying batch sizes, it may need shape buckets, padding, multiple graph instances, or eager fallback. Each choice has a cost: padding adds work, multiple captures consume setup time and memory, and fallback loses the replay advantage.

A defensible comparison protocol

Before enabling graphs in a service, collect evidence in a controlled sequence.

1. Identify the bottleneck. Use a GPU timeline or profiler to distinguish gaps between many short kernels from sustained device computation. A low utilization number alone is not proof of CPU launch overhead.
2. Hold the workload fixed. Compare eager and graphed paths on the same model, weights, dtype, inputs, batch distribution, sequence lengths, and device. Do not mix graph benefits with quantization or concurrency changes.
3. Verify correctness first. Compare eager and graphed outputs across several inputs, including boundary values for supported shape buckets. Use tolerances appropriate to the numerical precision.
4. Measure cold and warm separately. Report capture/initialization cost, first usable request, warmed p50/p95 latency, and GPU-side timing. CUDA is asynchronous, so synchronize correctly or use CUDA events where appropriate.
5. Account for extra memory. Record peak allocated/reserved memory, graph-pool retention, and the number of captured variants. A faster steady state that exhausts VRAM at realistic concurrency is not a successful deployment.
6. Test the actual request mix. Include varying sequence lengths, batch sizes, and graph misses. Measure how often the service enters an eager fallback or pays for new captures.

A useful results table has columns for graph mode, shape bucket, capture time, steady-state request latency, GPU time, peak memory, correctness result, and fallback frequency. Leave any field unknown rather than filling it with a vendor figure from a different setup.

Four failure patterns and what they mean

These are diagnostic categories, not definitive root causes. A CUDA error can also reflect an unrelated kernel or driver problem. Preserve the first useful error and a minimal reproducer before tuning memory settings.

CUDA graphs are not the same thing as torch.compile

The two technologies are often mentioned together but solve different problems. torch.compile traces and compiles PyTorch programs with guards and backend code generation; CUDA Graphs primarily package GPU work for low-overhead replay. Some compiler modes may integrate graph capture, but enabling one mechanism is not proof that the other is active or beneficial.

The distinction matters when diagnosing a regression. A recompilation caused by a shape guard is not the same event as capturing another CUDA graph. They can interact in a larger runtime, but should be measured separately. See RAMGPT’s torch.compile shape-guard guide for that adjacent mechanism.

Decision rule

Use CUDA graphs when a meaningful portion of your workload consists of repeated, graph-safe GPU work whose host submission overhead is measurable. Keep the captured region narrow enough to preserve correctness, then expand it only when profiling justifies the additional complexity. A valid conclusion is not simply ‘CUDA graphs are faster’; it is ‘for this request distribution and capture scope, replay saved enough host overhead to offset capture and memory costs.’

Primary sources

• PyTorch CUDA semantics: CUDA Graphs, constraints, and graph memory management
• NVIDIA CUDA Programming Guide: graph structure, instantiation, stream capture, and replay
• NVIDIA: Best Practices for PyTorch CUDA Graphs
• NVIDIA: Constant-Time Launch for Straight-Line CUDA Graphs