vLLM Tool Calling 400 or Empty tool_calls: Fix Parser and Template

If ordinary chat works but vLLM tool calling fails, use the response shape to choose the first fix:

• 400 requiring auto tool choice → start the server with the automatic tool-choice setting.
• 400 requiring a parser → configure the parser supported by the checkpoint.
• HTTP 200 but empty tool_calls → compare tool_choice, parser output, and the chat template.
• raw tool JSON/tags in content → parser and model output format do not match.
• bad function arguments → validate the schema and the model’s generated arguments.

Do not change GPU or model-loading settings until the raw /v1/chat/completions request reproduces the tool-calling failure directly.

Classify the symptom first

Inspect the HTTP status, redacted response body and the exact tool_choice value. A generic agent error is not enough evidence for a parser diagnosis.

Step 1: Verify the API and served model identity

The relevant endpoint is /v1/chat/completions. Obtain the model IDs that the running server advertises:

curl -sS http://127.0.0.1:8000/v1/models

Set the request’s model field to a returned ID. An invalid model ID may result in a 404 before the tool-calling code is even relevant. The vLLM served-model-name 404 guide covers that separate issue.

Step 2: Automatic tool choice requires a compatible parser

The current vLLM documentation lists two key server flags for tool_choice=auto:

• –enable-auto-tool-choice permits automatic tool selection.
• –tool-call-parser selects the extractor for model-generated calls.

A client cannot fix missing server startup flags merely by adding a tools array to its JSON request.

For Qwen2.5, the upstream documentation associates its tokenizer tool format with the hermes parser. A representative local launch command is:

vllm serve Qwen/Qwen2.5-7B-Instruct \
  --enable-auto-tool-choice \
  --tool-call-parser hermes \
  --host 127.0.0.1 \
  --port 8000

The command assumes an installed runtime capable of loading the checkpoint. It is not a performance measurement or guarantee of correct function selection. Check your installed version and supported flags with:

vllm serve --help

Do not use hermes indiscriminately. The vLLM guide documents different parser families for Llama, Mistral, Qwen, DeepSeek and other checkpoints. Use the parser and, where required, the chat template supported by the actual model revision.

Step 3: Reproduce the problem without an agent framework

With the example Qwen2.5 model serving successfully, send one complete minimal request:

curl -sS -i http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Qwen/Qwen2.5-7B-Instruct",
    "messages": [
      {"role": "user", "content": "Use the get_weather tool for Boston."}
    ],
    "tools": [
      {
        "type": "function",
        "function": {
          "name": "get_weather",
          "description": "Get the weather for a city.",
          "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string"}},
            "required": ["city"],
            "additionalProperties": false
          }
        }
      }
    ],
    "tool_choice": "auto",
    "max_tokens": 128
  }'

Use your actual /v1/models ID if different. The function declaration does not run a weather service or grant the model permission to execute code. It only describes a possible call.

For a tool call, inspect choices[0].message.tool_calls, the function name, and its JSON argument string. Also inspect finish_reason. An HTTP 200 response containing ordinary text is not automatically an API failure: auto permits the model to answer without using a tool.

To differentiate tool-selection behavior from parsing, try a named or required tool choice supported by your installed vLLM version. These modes apply different output constraints, so their results are informative controls, not interchangeable settings. The model can still choose poor argument values even when the output is syntactically constrained.

Step 4: Match the parser and chat template to the checkpoint

Tool-call parsing expects an output format. The chat template shapes the prompts for assistant messages, tool definitions and past tool results. If the configured parser assumes a different syntax than the checkpoint produces, you might see raw tool markup in ordinary content, missing tool calls, or partially parsed arguments.

Verify these items before changing unrelated server settings:

1. Actual checkpoint and tokenizer. A friendly served-model alias does not identify the underlying chat template. Record the resolved model ID, revision and tokenizer configuration.
2. Parser family. Compare the model to the official vLLM parser guide. New models may require a different parser, plugin or response-template mode.
3. Tool-aware chat template. Check whether the tokenizer’s existing template supports tools. Where required, use a documented –chat-template for the checkpoint.
4. Agent adapter. Compare the raw HTTP request with what the framework builds. Tool arrays, roles and tool_choice are sometimes transformed upstream of vLLM.

If direct curl works while the agent framework fails, isolate the client-side difference before changing vLLM itself.

Step 5: Valid JSON is not the same as a valid action

vLLM documents different levels of constraints for tool selection, function envelopes and argument schemas. Strictness support varies with parser and server options. Ordinary automatic parsing does not guarantee that every argument meets the intended schema or that the selected tool is appropriate.

The application must still validate and authorize each proposed action. A tool call requesting a command, file read or network operation is untrusted model output, even when the argument JSON parses successfully.

For failures, log a redacted version of the tool schema, message metadata, and generated arguments. Do not expose API credentials, private prompt histories, internal hostnames, or tool responses in a public issue.

Step 6: Isolate streaming and truncation

When stream=true, tool-call data can be spread across multiple events. A consumer that tries to JSON-parse each individual fragment may report a malformed call before the response is complete.

Compare an otherwise identical stream=false request. Inspect the token limit and stop conditions if the function output is cut off. The vLLM documentation discusses incomplete and truncated tool-call parsing, so a partial event or missing closer is not sufficient evidence that the underlying model or inference backend is broken.

Minimal evidence for an upstream report

Keep the installed vLLM version, actual model ID, redacted startup flags, tool parser, template choice, one minimal tool schema, tool_choice, HTTP status/body and whether streaming was enabled. Identify which boundary failed: configuration, model generation, parser extraction, streaming assembly, or tool execution.

The fast diagnostic order is 400: startup settings -> 200 without call: choice and parser/template -> call with bad arguments: schema and application validation.

Primary sources

• vLLM: Tool Calling and parser compatibility
• vLLM: serve CLI reference
• vLLM: request validation implementation

Also see vLLM 404: The Model Does Not Exist and How to Run an LLM Locally.