Constrained Decoding: How an LLM Can Follow a JSON Schema

AI Foundations #33 followed a request through prompt processing and token generation. Now consider a different problem: how can an application make the model’s output obey a machine-readable structure?

Suppose a workflow needs exactly this shape:

{"severity":"high","action":"review"}

Telling a model in the prompt to “return valid JSON” can work most of the time, but a probabilistic text generator may still produce an explanation before the object, miss a quotation mark or choose an unexpected field.

Constrained decoding changes which tokens may be selected during generation. Instead of merely asking for a format, the runtime can prevent many syntactically invalid continuations.

First: ordinary next-token selection

A language model produces scores, often called logits, for candidate tokens. A decoding algorithm transforms those scores into a choice.

model logits
     |
     v
sampling policy: greedy, temperature, top-p, etc.
     |
     v
chosen token
     |
     v
append token and repeat

Without additional restrictions, the model can generate any token its vocabulary and sampler permit. Prompt instructions influence probabilities; they do not create a hard grammar guarantee.

Add a grammar to the selection process

With constrained decoding, the runtime also tracks what the output has generated so far and what the specified grammar or schema allows next.

The grammar and logits meet before sampling. Real tokenizer IDs can span multiple characters, and the state advances after each selected token.

Imagine the output begins with an object-opening brace. A grammar may now permit a quoted property name, optional whitespace or a closing brace, depending on the schema. It should not permit arbitrary prose at that position.

A practical implementation does not need to teach the model JSON syntax from scratch. It can constrain the sampler according to the allowed continuation.

But tokens are not individual characters

A tokenizer might represent several characters, spaces and pieces of words in one token. One token could include the end of a quoted string and the beginning of another syntactic construct.

That complicates masking. A grammar operates over output structure, while the model chooses token IDs.

A correct implementation must account for what character sequences candidate tokens decode to, not merely whether a token’s first character is allowed. Research on automata-based constrained decoding studies this tokenizer/grammar alignment problem directly.

That is why “just filter illegal characters” is not a sufficient description of a production constrained decoder.

JSON Schema is a specification, not necessarily a complete implementation

JSON Schema can describe required properties, types, enumerated choices and other structural rules. Consider an incident-routing response:

{
  "type": "object",
  "properties": {
    "severity": {
      "type": "string",
      "enum": ["low", "medium", "high"]
    },
    "action": {
      "type": "string",
      "enum": ["review", "escalate"]
    }
  },
  "required": ["severity", "action"],
  "additionalProperties": false
}

In principle, the schema narrows the legal surface: severity cannot become “extremely urgent” and action cannot become a paragraph.

In practice, not every decoding backend implements every JSON Schema keyword or every interaction between keywords. Some systems translate supported schema features into grammars; others use specialized state machines. Verify the exact supported subset in your selected runtime and version.

The llama.cpp GBNF guide describes its grammar format and JSON Schema conversion with limitations. The vLLM structured-output documentation documents schema, choice, regular-expression and grammar options, along with backend configuration.

Three levels of correctness

It helps to distinguish three questions:

A perfectly valid JSON object can contain the wrong answer. A valid incident label is not proof that the model examined the right logs or understood the event.

Likewise, a grammar should not be treated as a trust boundary for authorization. Application code must independently validate the generated payload and decide which downstream actions are permitted.

Why grammar design affects speed

The model still performs inference, but a constrained sampler does extra work: prepare or compile the rule set, evaluate allowed tokens and update its grammar state as tokens are emitted.

The overhead depends on grammar complexity, token vocabulary, backend, and whether compiled constraints can be reused. A complex schema may have different latency behavior from a simple two-choice output.

A fair comparison therefore measures both:

• First-response overhead, including grammar compilation or initialization.
• Steady-state generation, using the same model, prompt, output length and cache state.

A model generating shorter JSON can appear “faster” in total response time simply because it outputs fewer tokens. That does not isolate the grammar implementation.

When to use constrained decoding

It is valuable when a downstream system expects fields, enums, lists or exact formats and a parse failure is costly.

For a small classification task, a fixed choice can be simpler than a large JSON Schema. For a structured extraction request, a supported schema can be appropriate. For a specialized language, a grammar may be the natural representation.

It is less useful as a substitute for correctness evaluation, model calibration, access control or an evidence-based review.

A minimal verification plan

1. Pin the runtime version and specify which structured-output backend is active.
2. Test positive cases with typical and edge-case content.
3. Probe escaped quotes, Unicode, empty collections and allowed null values where applicable.
4. Check required fields, unknown keys, enums and nested combinations against a separate JSON Schema validator.
5. Test refusal and maximum-token truncation behavior. An interrupted generation may be incomplete even with a grammar.
6. Treat all returned values as untrusted data before invoking tools or changing real systems.

The key distinction is: a schema can constrain which strings the model is allowed to produce; it cannot guarantee that those strings describe the world accurately.

Primary sources

• vLLM structured outputs documentation
• llama.cpp GBNF and JSON Schema guide
• Automata-based constraints for language model decoding