vLLM 0.31 Admission Control: Active Sequences vs Queue Limits

When an LLM server has too many simultaneous decodes, reducing the batch size sounds simple. In vLLM, though, a traditional concurrency setting also influences how much runner state is allocated. A new admission-control option separates these two decisions. The engineering question is not just how many requests can be processed: which requests are allowed into RUNNING, and which still wait?

vLLM 0.31.0, released October 5, 2026, includes the scheduler change introduced by PR #56758. It adds max_num_active_seqs alongside max_num_seqs. This is a release-specific, source-based explanation, not an independently measured speedup.

The three limits govern different boundaries

The vLLM serve CLI reference documents these boundaries. The key distinction is that max_num_active_seqs limits the WAITING-to-RUNNING transition inside an engine. The separate queue cap can return HTTP 503 when the API server reaches its configured limit. The queue cap is enforced across the data-parallel ranks served by that API-server process; do not multiply per-rank observations into an imaginary universal engine limit.

Without an explicit active-sequence cap, vLLM uses max_num_seqs as the default admission bound. The new cap must be no greater than max_num_seqs.

A concrete example, not a benchmark

Consider a deployment with runner capacity 128 and an admission cap of 16:

vllm serve facebook/opt-125m \
  --host 127.0.0.1 --port 8000 \
  --max-num-seqs 128 \
  --max-num-active-seqs 16

This is an illustrative configuration, not an assertion that this specific model and hardware were tested here. The command assumes vLLM 0.31.0 or a compatible later build; confirm the model, backend and installed CLI before using it in your environment.

Suppose 24 independent requests are ready. Ignoring other scheduler and KV-memory constraints, the active cap permits at most 16 requests to occupy RUNNING; the remaining requests wait for opportunities to enter. Runner buffers and CUDA graph planning still follow 128. A smaller active cap does not reserve 112 requests’ worth of extra throughput, and it does not guarantee that 16 fit in available KV memory.

The upstream implementation PR counts requests waiting for streaming input toward the admission threshold because they retain relevant runner slots. Lowering the cap is also not a preemption command: existing requests complete, then the lower admission target takes effect.

Why changing the wrong flag misleads operators

An operator seeing a rising p95 time to first token may be tempted to lower every concurrency limit. But each change targets a different mechanism:

1. Busy RUNNING set: A lower active cap may reduce decode contention or scheduling overhead, but it also leaves more requests waiting. End-to-end p95 can get worse if queue time dominates.
2. Excessive HTTP backlog: A finite API-server queue cap provides a rejection boundary, making overload visible as 503 instead of allowing unlimited waiting. Clients need an explicit retry or load-shedding policy.
3. Runner memory or graph overhead: Lowering max_num_active_seqs alone does not shrink static runner buffers or the CUDA graph capacity built for max_num_seqs. For that question, investigate the runner setting separately.
4. KV-cache exhaustion: Admission is still subject to memory and token-budget constraints. The active cap is not an allocation guarantee.

These are hypotheses to test, not claims that a particular cap improves performance. For background on the model-serving trade-off, see AI Foundations #28: Continuous Batching and Scheduling.

Verify version support before tuning

Record the installed version and the actual help output:

python -m pip show vllm
vllm serve --help | grep -E 'max-num-(seqs|active-seqs|queued-reqs)'

If max-num-active-seqs is absent, do not assume the behavior of the 0.31 release applies to a different wheel or older deployment. Keep a record of your image digest or package build. The 0.31.0 release notes and merged scheduler PR establish the initial version context; the latest online CLI page may describe newer behavior than a pinned installation.

A startup validation failure is possible if the active cap exceeds max_num_seqs. Confirm both values together rather than editing flags independently.

An A/B procedure that can reveal the trade-off

Use the same model artifact, image digest, GPU allocation, sampling policy, request arrival process, prompt-length distribution and output limits in both runs. Warm each configuration using the same procedure and keep prefix-cache state comparable.

The values above are test inputs, not measured throughput or recommended defaults. Sample at a fixed arrival rate first, then repeat near saturation. Report requests/sec and total output tokens/sec alongside p50/p95 queue delay, time to first token, inter-token latency, rejections and GPU/KV utilization. Keep rejected requests in the denominator when reporting service outcomes.

vLLM exposes server-wide Prometheus metrics, while its per-request metrics guide explains why queue time should not be collapsed into decode time. You should also check whether a benchmark client’s own request queue is masking the server’s queue.

Decision rule

Use max_num_active_seqs when the research question is whether a lower admitted decode concurrency helps a fixed runner configuration. Use max_num_queued_reqs when the question is how to reject excess demand at the API boundary. Use max_num_seqs when runner capacity itself is the intended variable. Do not describe any of them as a universal latency optimization without controlled measurements.

Primary references

• vLLM v0.31.0 release notes
• Merged scheduler PR #56758 and its test plan
• vLLM serve CLI: scheduling and queue limits
• vLLM production metrics