vLLM on GB10: Investigating an FP8 cuDNN GEMM Hang
An October 10 vLLM report traces a long-running DGX Spark inference freeze to a cuDNN FP8 GEMM path. Symptoms, control tests, and an attributed kernel-selection workaround.
Approximately 5 min read
A running inference server can be unhealthy even while its HTTP API still responds.
An October 10 vLLM issue describes an NVIDIA DGX Spark (GB10, SM 12.1) system that stops generating tokens after sustained serving load. The reporter did not merely observe a timeout: they collected engine counters, a Python stack, a CUDA core dump and a comparison against a different FP8 GEMM implementation.
The actionable observation is narrow: in that environment, disabling the FlashInfer FP8 scaled-matrix kernel changed the selected implementation to CUTLASS and prevented the reported freezes through substantially longer tests. This is an upstream-reported workaround, not a universally verified cuDNN bug fix.
The failure signature
The affected deployment used an NVIDIA NVFP4 model that also has FP8 linear operations: nvidia/Qwen3.6-35B-A3B-NVFP4. The issue identifies a vLLM nightly container, CUDA 13.0, cuDNN 9.20 and FlashInfer 0.6.16.post3.
Four unmodified boots reportedly stopped progressing between 237,065 and 287,155 engine steps. Three reached that range in about 1.4 hours under the author’s two-request soak workload. Another system ran light traffic for days and later stopped after a benchmark raised the cumulative step count.
At the freeze:
- Token-generation and iteration-token counters stopped advancing.
- One request remained in the running state, but clients timed out.
- The API process still answered requests such as /metrics and /v1/models.
- GPU utilisation appeared high, but power draw was unusually low for useful inference work.
- There was no matching NVIDIA Xid error in the reporter’s logs.
That combination should change the investigation. A successful health endpoint does not prove that an engine is making forward progress. Likewise, high GPU utilisation alone does not demonstrate useful token generation.
Why the GPU evidence matters
The engine-side Python stack ended at a CUDA stream synchronization call, waiting for GPU work to finish. The reporter then inspected a CUDA core dump.
They found a single resident cuDNN-generated FP8 matrix-multiplication kernel. Its warps appeared to be waiting repeatedly on barriers. This points away from an ordinary HTTP stall and toward a GPU-side synchronization problem.
However, a blocked barrier is a symptom, not complete proof of the original defect. The report does not establish whether the underlying problem is exclusively in cuDNN, a backend dispatch decision, a combination of input shapes, or some other prerequisite. The author explicitly had not created a standalone minimal reproducer.
The reported selection path was:
vLLM ModelOptFp8LinearMethod
|
v
FlashInferFP8ScaledMMLinearKernel
|
v
flashinfer.bmm_fp8(backend="auto")
|
v
cuDNN-generated SM120-family GEMM
A kernel selected automatically for a supported architecture is not necessarily the best choice for every shape and sustained workload. The selected backend is an important part of the experiment’s identity.
The one-variable workaround
The reporter set:
export VLLM_DISABLED_KERNELS=FlashInferFP8ScaledMMLinearKernel
After restarting vLLM, its initialization log showed CutlassFP8ScaledMMLinearKernel instead.
This is not an instruction to change every vLLM deployment. It is a diagnostic workaround for the specific hardware, version and quantized model path identified above. Confirm the new selected kernel in the log; setting the environment variable alone is not sufficient evidence that a different kernel was used.
The upstream author’s two reported control runs were:
| Configuration | Reporter-observed result |
|---|---|
| Default FlashInfer FP8 kernel | Four boots eventually froze, around 237K–287K engine steps |
| FlashInfer FP8 kernel disabled; CUTLASS selected | One six-hour soak completed 9,323 requests with no freeze |
| Same workaround, separate longer run | About 9.3 hours, 9,811 requests in the soak phase; at least 1.77M total engine steps without a freeze |
These figures come from issue #61007. They are not RAMGPT results and do not establish statistical reliability across other GPUs. Four short greedy answers matched; one longer answer differed by a word. This is insufficient for a formal numerical-equivalence claim.
A diagnostic checklist for operators
Start with evidence collection, not a blind driver upgrade.
- Identify the execution path. Record GPU architecture, driver, CUDA, cuDNN, FlashInfer, model checkpoint, quantization mode and exact vLLM image or commit. Preserve initialization lines naming the selected FP8 linear kernel.
- Separate API liveness from engine progress. Sample completed-request counts and generated-token totals at intervals. A responsive /metrics endpoint with flat token counters is a warning sign.
- Capture the first stuck request. Record timestamp, request concurrency, prompt/output lengths and cumulative iterations if exposed. Avoid including sensitive prompts in shared logs.
- Compare a controlled restart. If the reported environment matches, consider the kernel-disable variable. Keep the model, workload, concurrency and memory flags unchanged while verifying the alternate kernel was selected.
- Run a bounded soak test. Compare completed requests, errors, token counters and numerical sanity. A quick five-prompt smoke test cannot rule out a failure that appeared only after hundreds of thousands of steps.
A useful paired log includes the same workload and version on both sides of the single kernel-selection change. Do not compare a new model plus a driver update against an old engine and attribute success to one flag.
Related upstream work does not equal a merged fix
The issue points to vLLM PR #48210, PR #57209 and FlashInfer issue #3566. Those discussions concern backend selection and cuDNN behavior on newer NVIDIA architectures.
They support investigating the dispatch path; they do not prove that a particular open PR has already resolved this hang. The reporter tested the CUTLASS selection workaround, not a proposed cuBLAS pin.
What this teaches about inference reliability
There are at least three distinct health questions:
- Can the server answer an HTTP request?
- Is the engine scheduler making progress?
- Are GPU kernels returning and producing valid tokens?
Those are different failure domains. A production monitor that checks only the first can call a serving process healthy while the second and third are broken.
For the operating model behind this distinction, see AI Foundations #33 on prefill and decode. For an independent vLLM cache-state diagnosis, see the earlier NVFP4/DFlash case study.
Status, October 10, 2026: the upstream report is open. The workaround has substantial reporter-provided soak evidence, but a root-cause fix and broad cross-platform validation remain outstanding.
Primary sources
- vLLM issue #61007: GB10 FP8 GEMM hang and controlled workaround
- vLLM PR #48210: related backend-selection discussion
- vLLM PR #57209: related backend-selection discussion
- FlashInfer issue #3566: cuDNN FP8 backend symptoms