vLLM DeepGEMM 'Runtime Is Already Registered': Update Past the Fix

If vLLM fails during a FlashInfer MoE startup path with:

ImportError: generic_type: type "Runtime" is already registered!

update to a build that contains vLLM PR #48962 (merge commit cc73cca9c88bc19a368a910a630e9c5caec61ac3) before reinstalling pybind11 or randomly changing model packages.

Upstream issue #59904 was closed on October 11 after the reporter rechecked current main: PR #48962, merged October 6, moved vendored DeepGEMM from the duplicate pybind11 registration path to TORCH_LIBRARY. The reporter’s import repro and FlashInfer MoE tests then passed.

The original failure loaded vendored DeepGEMM once under vLLM’s internal namespace and again through deep_gemm.*, causing the native Runtime type to be registered twice.

The affected path reported upstream

Issue #59904 reports the failure with:

vllm/vllm-openai:nightly
flashinfer-python==0.7.0.post1
vendored vllm.third_party.deep_gemm
--moe-backend flashinfer_moe_ep_mega_deep_gemm

The reporter used four GB300 GPUs.

The exact environment matters because this is not a generic “all vLLM + DeepGEMM combinations fail” bug.

The default path in the same report could run because pre-quantized weights and the fused stage avoided the import sequence that triggers the collision.

Why the error says Runtime is already registered

DeepGEMM includes a compiled pybind11 extension.

pybind11 registers C++ types with Python when the extension module is initialized.

Loading the same extension once is normal:

module load
-> register Runtime
-> continue

Loading it again under a second module identity can attempt:

second module load
-> register Runtime again
-> pybind11 rejects duplicate registration

That is where the message comes from.

The problem is not that two requests created two runtime objects.

It is that Python imported the same native extension through two namespace paths.

The aliasing problem

vLLM includes a helper that exposes its vendored DeepGEMM package to FlashInfer.

The reported implementation aliases the top-level package:

vllm.third_party.deep_gemm
        |
        +--> visible as deep_gemm

But aliasing only the top-level package does not automatically guarantee that every already-loaded submodule is also aliased.

When FlashInfer later imports:

from deep_gemm.utils import ...

Python can treat deep_gemm.utils as a new module name.

That module then reaches the compiled extension through the new namespace and can initialize it a second time.

Why the crash may be conditional

The reporter identified two FlashInfer paths that can import the affected utility module:

• a weight-quantization path when incoming weights are not already pre-quantized;
• an activation-staging fallback when the fused stage is disabled or unsupported.

That explains a confusing operational pattern:

default configuration -> starts
fallback configuration -> crashes
same image, same model family

The error is therefore sensitive to which code path is exercised, not merely whether FlashInfer and DeepGEMM are installed.

Confirm you have the same failure before changing packages

Start with the traceback.

The relevant pattern is:

vllm/third_party/deep_gemm/utils/...
from .._C import ...
ImportError: generic_type: type "Runtime" is already registered!

Then record:

python - <<'PY'
import vllm
import flashinfer
print("vllm", vllm.__version__)
print("flashinfer", getattr(flashinfer, "__version__", "unknown"))
PY

Also capture:

• the vLLM image or exact commit;
• FlashInfer version;
• whether a separate site-packages deep_gemm is installed;
• selected MoE backend;
• relevant FlashInfer staging environment variables.

Do not assume a random Python reinstall is the fix until the namespace pattern matches.

Capture the imports in the failing process

A new Python process that only imports sys cannot show the modules loaded by the server. Capture the namespace in the process that runs the reproducer.

Save this wrapper as inspect_imports.py:

import runpy
import sys
from pathlib import Path

if len(sys.argv) < 2:
    raise SystemExit("Usage: python inspect_imports.py reproduce.py [arguments]")

target = Path(sys.argv[1]).resolve()
sys.argv = [str(target), *sys.argv[2:]]
sys.path.insert(0, str(target.parent))
try:
    runpy.run_path(str(target), run_name="__main__")
finally:
    for name, module in sorted(list(sys.modules.items())):
        if "deep_gemm" in name:
            print(name, getattr(module, "__file__", None), file=sys.stderr)

Run it with the Python script that reproduces your failing initialization:

python inspect_imports.py reproduce.py

Look for both vendored and deep_gemm.* names pointing into the same extension tree. Python may remove a failing module from sys.modules, so retain the original traceback alongside this output.

Why reinstalling pybind11 is unlikely to solve this root cause

The exception comes from pybind11, but the reported defect is module exposure/aliasing.

Changing pybind11 versions may alter behavior, but it does not fix the fundamental situation where one native extension is initialized twice under different import names.

Treat the import graph as the primary root cause unless your traceback points elsewhere.

Practical mitigations

The safest short-term option is to avoid the exact fallback path that triggers the duplicate import when your workload supports that.

The upstream report says the default fused path worked in its configuration.

The upstream fix is now on main. Use a package, image, or commit that contains PR #48962; do not assume an older pinned image was rebuilt with it.

Avoid manually patching sys.modules in production unless you understand the import ordering. A partial alias can reproduce the same class of bug under a different submodule.

The upstream aliasing proposal

Issue #59904 was closed on October 11 because PR #48962 had already removed the duplicate pybind11 registration mechanism on main. The earlier aliasing proposal explains the failure, but the merged fix changes the extension binding path itself.

The intended result is:

vllm.third_party.deep_gemm.utils
and
deep_gemm.utils

-> same Python module object
-> native extension initialized once

That is a much more targeted fix than suppressing the pybind11 exception.

Fast diagnosis checklist

For:

generic_type: type "Runtime" is already registered!

on a vLLM FlashInfer/DeepGEMM MoE path:

1. confirm the traceback passes through vendored DeepGEMM;
2. record exact vLLM and FlashInfer versions;
3. check whether a separate deep_gemm installation also exists;
4. inspect sys.modules for both vendored and aliased module names;
5. determine whether a fallback staging/quantization path triggers the import;
6. avoid the affected path or move to a revision with the upstream aliasing fix when available.

The important distinction is:

this is a duplicate native-module initialization problem, not an ordinary duplicate Python object.