vLLM Per-Request Metrics: Separate Queue, Prefill, and Decode

A server can report respectable aggregate tokens per second while some users still wait a long time for the first token. The useful question is not merely whether the GPU is busy; it is where each request spends its time.

The current vLLM documentation describes an optional response-level metrics object that helps separate scheduler queueing, first-token latency, and subsequent generation. This article examines the instrumentation boundary, its blind spots, and a reproducible evaluation plan. It is a source-based engineering guide, not a claim that a particular deployment has been benchmarked.

Three clocks, not one speed number

Consider a request competing with several others:

request arrives
      |
      v
wait for scheduler admission and capacity
      |
      v
prefill / prepare prompt state
      |
      v
first generated token
      |
      v
more output tokens ... final token

These stages consume different resources and suggest different remedies. A large prompt can increase work before the first token. Heavy concurrency can create queueing even when every individual request is short. Slow inter-token delivery can indicate issues in decode scheduling, memory bandwidth, model size, or competing work.

AI Foundations #27 introduced latency and throughput. The operational extension is to capture per-request evidence, rather than infer every user’s experience from one process-wide gauge.

What vLLM can return for a request

The official vLLM Per-Request Metrics guide documents a server flag:

vllm serve unsloth/Llama-3.2-1B-Instruct \
  --enable-per-request-metrics \
  --host 127.0.0.1 --port 8000

This is an illustrative invocation. Model availability, hardware compatibility, and flag support depend on the installed build; check your installed version and help before running the example.

For a supported single-generation API response, the optional metrics object can include the following fields:

The distinction in the final row matters: tokens_per_second includes the prefill/first-token interval, so it is not simply the reciprocal of mean inter-token latency. Also, the documented time-to-first-token metric begins at scheduling; it does not include the separate queue wait. For complete user-visible latency, client timing and network overhead are additional factors.

Read the fields as a diagnostic matrix

Suppose two requests have different timing profiles. These numbers are illustrative, not measured results:

A has a much longer queue wait; B has a much longer interval between scheduling and its first generated token. Blaming both on decode performance would direct investigation toward the wrong subsystem.

Use controlled experiments to evaluate competing explanations. For queueing, compare light-load and high-load distributions while holding prompt/output lengths reasonably stable. For slow TTFT, compare similar batch/concurrency conditions at several prompt lengths and prefix-cache states. For slow generation, examine the model and serving configuration, active sequences, decoding parameters and GPU contention.

A single response’s fields are descriptive timings. They do not prove which kernel, cache miss, network hop or scheduler decision caused the delay.

The streaming trap: inspect the final usage event

Non-streaming clients that preserve unknown response fields can read the response-level metrics object directly. In a streaming completion, vLLM documents attaching the metrics to the final usage chunk, not each intermediate text delta.

A diagnostic API request can opt into usage reporting:

{
  "model": "unsloth/Llama-3.2-1B-Instruct",
  "messages": [{"role": "user", "content": "Explain prefix caching in three sentences."}],
  "stream": true,
  "stream_options": {"include_usage": true}
}

The server must already be launched with per-request metrics enabled. The client also has to retain and inspect that final event. An application that stops reading the stream after the last visible text fragment may silently discard the metrics.

The documentation also describes a server-side option to force usage inclusion. When debugging an SDK integration, first compare the raw HTTP response with what the client library exposes instead of assuming that missing metrics mean missing measurements.

When metrics are intentionally unavailable

It is incorrect to assume that every HTTP 200 response will contain timing values.

The documented single-generation attribution restriction suppresses timing metrics when a chat completion asks for n greater than 1; separate completions API requests containing multiple prompts have a similar limitation. For the Responses API, workflows with multiple model-generation turns can also suppress the aggregate response-level timing object, because one timing record cannot correctly describe several turns.

For a single-token output, mean inter-token latency has no interval to average, so it may be null. Fields can also be null when the underlying timing information is unavailable. Finally, per-request statistics depend on server-side stats logging: vLLM rejects the per-request flag when statistics logging is explicitly disabled.

Null is not zero. Excluding null records without understanding why can bias performance reports toward simpler requests.

Per-request timing and Prometheus solve different questions

The production vLLM metrics reference documents a server-level Prometheus endpoint:

curl -sS http://127.0.0.1:8000/metrics

That endpoint exposes aggregate counters, gauges and histograms, including workload-level latency distributions and request processing signals. It helps answer questions such as whether queueing increased across a fleet or whether a deployment’s tail latency changed over time.

Response-level fields answer a narrower question: what happened to this attributable request? The two views complement rather than replace each other.

A useful evidence record joins request-level timings to a small set of safe metadata: prompt token count, generated token count, streaming mode, request classification, model version and concurrency interval. It should not include full private prompts or unrelated logs. When aggregating comparisons, partition by meaningful workload characteristics rather than hiding short queries and long-context requests in one mean.

Instrumentation has a cost

vLLM’s own documentation warns that per-request metric computation can add noticeable CPU overhead at high concurrency. Instrumentation should therefore be benchmarked itself.

An evaluation plan can compare the same workload with the feature disabled and enabled, holding the model, hardware, prompt length distribution, output length distribution, arrival rate, warmup and sampling policy as constant as practical.

Observe not just the metric object’s values but also completed requests per second, CPU pressure, tail latency, and any requests rejected or cancelled. A timing feature is not free merely because the overhead is incurred outside GPU kernels.

A disciplined rollout checklist

1. Verify the installed vLLM version exposes the desired flag, then enable it for a bounded diagnostic workload.
2. Obtain a single-request, non-streaming response and inspect each field’s units and nullability.
3. Exercise streaming with final usage reporting; ensure the client consumes the terminal event.
4. Test a request with multiple output sequences and record whether timing fields are suppressed.
5. Compare aggregate Prometheus histograms against a properly sampled set of response-level timings.
6. Benchmark telemetry overhead at the concurrency relevant to the deployment; do not extrapolate a low-load result.
7. Store sanitized metrics rather than raw prompts, credentials or infrastructure identifiers.

The engineering takeaway is simple: queue time, time to first token and decode behavior are different clocks. Measure the clock associated with the complaint before changing GPU settings or concluding that a model is inherently slow.

Primary sources

• vLLM: Per-Request Metrics
• vLLM: Production Metrics and Prometheus endpoint
• vLLM: serve CLI reference

For the conceptual background, continue with AI Foundations #27: Latency and Throughput and AI Foundations #33: Disaggregated Prefill and Decode.