Structured outputs: does forcing JSON make a model worse at thinking?

Barakaeli Lawuo, Apr 22, 2026

In 2024 a team from Appier AI Research and National Taiwan University asked GPT-3.5 Turbo to take the last letter of each of four words and join them. Answering in plain text, with room to reason step by step, it got 56.7% exact match. Told to answer in a JSON format described in the prompt, it got 25.2%. With JSON mode switched on, the API flag that makes the output valid JSON, it got 1.78%.¹ The same pattern held on GSM8K, the grade-school math benchmark: 76.6% in plain text, 49.3% with the JSON instruction, 29.87% in JSON mode.¹

The paper, titled "Let Me Speak Freely?", concluded that format restrictions cause "a significant decline in LLMs' reasoning abilities," and that stricter constraints generally hurt more.¹ Its Figure 1 is a single GSM8K question about overtime pay. In plain text the model works through the steps and answers 460, which is correct. Asked for JSON, it writes a one-sentence plan in the reasoning field and puts 490 in the answer field.¹

Anyone who ships model output into a parser has a stake in whether this is true. The rest of this post takes the claim apart: how format constraints are enforced, what they cost, the evidence that they help, and the rebuttal that says the 2024 result came from the prompts rather than the constraint.

Three ways to ask for JSON

Tam et al. compared three levels of strictness, and the distinction matters for everything that follows.¹

Constrained decoding

the decoder is only allowed to pick tokens that keep the output valid under some formal rule, such as a regular expression, a grammar, or a JSON Schema. Invalid tokens are removed before sampling.⁴

JSON mode

a provider API flag (OpenAI and Gemini offered one) that makes the output valid JSON. Tam et al. write that they assume it works like the constrained decoding in Outlines, which is an assumption, not a documented fact.¹

Format-restricting instruction (FRI)

the prompt asks for JSON, XML or YAML in a given shape, but nothing enforces it at decode time.¹

NL-to-Format

two calls: the model answers in natural language first, then a second step converts that answer into the target format.¹

JSON Schema

a language for describing the shape of JSON data: field types, required keys, string patterns, array lengths. Most constrained decoding tools now accept it as input.⁴

Every schema in the study had exactly two fields, one for reasoning and one for the answer. The authors kept it that simple on purpose, to isolate the effect of format on reasoning.¹ Their results split by task type. On reasoning tasks (GSM8K, Last Letter, Shuffled Objects) looser formats did better. On classification tasks the opposite happened. Gemini 1.5 Flash went from 41.6% in text to 60.3% in JSON on DDXPlus, a 49-way medical diagnosis task. The authors guessed that a restricted answer space cuts down on answer-selection mistakes.¹

They also found something that turns out to matter a great deal later. On Last Letter, 100% of GPT-3.5 Turbo's JSON-mode responses put the "answer" key before the "reason" key.¹ The model committed to an answer first and wrote its reasoning afterwards. That is direct answering, not chain of thought, whatever the field names say.

What the decoder actually does

A language model picks one token at a time. At each step it produces a score, called a logit, for every token in its vocabulary, and the next token is sampled from those scores. Brandon Willard and Rémi Louf, in the paper behind the Outlines library, write the constrained version like this:²

$\tilde{S}_t$ is the sequence of $t$ tokens generated so far. $\theta$ is the model's trained weights. $\alpha$ is the vector of scores, one per vocabulary entry, so it has $N$ entries, where $N$ is the vocabulary size and is often $10^4$ or more.² $m(\tilde{S}_t)$ is a boolean mask of the same length: 1 for tokens that keep the output valid given what has already been written, 0 for tokens that would break it. $\odot$ multiplies element by element, so every forbidden token is switched off. The last line samples the next token $\tilde{s}_{t+1}$ from what is left. The model's preferences among the valid tokens stay as they were. Only the support of the distribution shrinks.²

The expensive part is $m$. The simple way to build it is to check every vocabulary entry against the constraint at every step, which is a fixed $O(N)$ cost per generated token, and if matching restarts from the beginning of the output each time, that cost grows as the output gets longer.² Willard and Louf's move was to convert the regular expression into a finite-state machine (a set of states with transitions that consume characters) and precompute, for every state, which vocabulary strings the machine can accept from there. At generation time the decoder only has to track which state it is in and look up the allowed set in a hash map, which costs $O(1)$ on average.²

The figure shows why the index has to cover every state and not just the start. A token like ".2" can match the middle of the pattern, so the lookup table needs an entry for each state it might be consumed from.² For JSON, SQL or Python, a regular expression is not enough, and the paper extends the same indexing to context-free grammars using pushdown automata, which add a stack to handle nesting. Their index for a slightly augmented Python grammar came to about 50 MB, even built without removing redundant states.²

The only cost measurement in the paper is a head-to-head against Guidance, which at the time re-ran partial regex matching from the start of the output and scanned GPT-2's 50,257-token vocabulary on every step. In their plot, Guidance took close to two minutes to generate 60 tokens, while Outlines stayed near zero out to 100 tokens.² Each point is one timed run, and the authors add the caveat "barring any configuration oversights."² Guidance works differently now: in a 2025 benchmark, below, it computed constraints on the fly and was the fastest engine tested.⁴

Grammar-constrained decoding, as Geng et al. called it in 2023, uses an incremental parser to supply the set of allowed next tokens and can plug into greedy decoding, beam search or sampling.³ Their latency numbers show where the overhead comes from. Pure decoding on an A100 took 54 ms per token for LLaMA-7B. The grammar step, run on a consumer CPU, added 1 ms per token for entity disambiguation and 4 ms for constituency parsing, but 69 ms for an extraction grammar with millions of rules.³ The overhead depends on how big the grammar is. For small grammars it is close to nothing.

The case for constraining

The first argument for constraints is the obvious one: without them, models often produce output that does not parse. Geng et al. asked few-shot LLaMA models, without finetuning, to produce Penn Treebank parse trees. Unconstrained LLaMA-33B produced a valid tree 64.2% of the time. With an input-dependent grammar, one built for each sentence so that it has to include every word of the input, validity was 100%, and bracketing F1 went from 42.9 to 54.6.³ On entity disambiguation, where the answer must be one of a list of candidate entities, LLaMA-33B's average accuracy across six datasets rose from 54.1 unconstrained to 80.3 constrained.³

JSONSchemaBench, from EPFL and Microsoft, tested the same idea at scale with about 10,000 real JSON schemas from GitHub, Kubernetes configs, API specs and a function-calling dataset.⁴ It separates two numbers. Declared coverage is the share of schemas a framework accepts without erroring. Compliance rate is the share of accepted schemas where the output actually validated. For open engines the model was Llama-3.2-1B-Instruct with greedy decoding.⁴

With no constraint, a 1B model produced valid JSON for 38% of medium schemas and 13% of hard ones.⁴ Constraints raised that a lot, but no open engine reached 100% on these two sets, and they failed in different ways. Outlines lost compliance mostly to timeouts: features such as minItems, maxItems and enum could take from 40 seconds to 10 minutes to compile.⁴ The closed providers showed the opposite pattern. OpenAI's compliance was near perfect on what it accepted but it accepted very little, which the authors read as a deliberate choice to support only the schema features they can guarantee.⁴ On speed, Guidance ran at 6.37 ms per output token on the function-calling set against 15.40 ms for the unconstrained model, because it skips ahead over tokens the grammar forces. Outlines ran at 30.33 ms and spent 3.48 seconds compiling first.⁴

The rebuttal: same model, same prompt

Will Kurt of .txt, a company that builds structured generation tools, published a response in 2024. It is a company blog post, not a peer-reviewed paper, and the author has a commercial stake, so weigh it accordingly. It includes reproduction notebooks, and JSONSchemaBench later adopted its setup.^5,4

Kurt's first point concerned the scoring. Tam et al. did not score answers with a fixed regex. They used claude-3-haiku as an answer extractor and called it a "perfect text parser."¹ Rescoring the paper's own saved Llama-3-8B Last Letter outputs, Kurt got 0.35 with a strict regex, 0.57 with the AI parser, and 0.61 with four hand-written regexes.⁵ His second point concerned the prompts. The saved JSON-mode prompt he quotes says "You must use the tool" but never mentions JSON, gives no schema and names no tool.⁵ The paper's plain-text prompts described the expected answer format explicitly. His third point was that JSON mode is not the same thing as grammar-constrained generation.⁵

He then reran the three reasoning tasks with Llama-3-8B-Instruct, using the same prompt for both conditions: a system message that shows the schema, a worked example in that schema with reasoning before the answer, and the chat template applied properly. Structured generation came out slightly ahead on every task: 0.78 against 0.77 on GSM8K, 0.77 against 0.73 on Last Letter, and 0.44 against 0.41 on Shuffled Objects.⁵

JSONSchemaBench then ran the same three tasks with Llama-3.1-8B-Instruct, following Kurt's prompts. Every constrained engine beat the unconstrained model on every task, by margins of up to about 4 points.⁴

Tam et al.'s own appendix is not as one-sided as their abstract. When they tested gpt-4o-mini with OpenAI's newer schema-enforced structured outputs, it scored 91.71 on GSM8K against 86.95 for JSON mode and 87.17 for JSON by instruction, and it scored best of all four formats on Last Letter at 86.07. Plain text still won on GSM8K at 94.57.¹ The authors also found that parse failures were not the main explanation for the gaps. Their example is LLaMA 3 8B, whose Last Letter output failed to parse only 0.148% of the time in JSON while scoring far below plain text.¹

The condition both sides meet on: reasoning goes first

The two sides disagree less than their headlines suggest. The worst numbers in Tam et al. come from a condition where the model put the answer before the reasoning, or never had room to reason.¹ The better numbers from the rebuttals come from setups that force reasoning to come first and constrain only the answer.^5,4 The schema that makes this happen is small:

{
  "type": "object",
  "properties": {
    "reasoning": { "type": "string" },
    "answer":    { "type": "string", "pattern": "^[a-z]+
quot; }
  },
  "required": ["reasoning", "answer"]
}

One catch: a JSON Schema's list of properties does not fix the order they are generated in. Whether "reasoning" gets written first depends on the engine and on what the prompt and examples show. That is the author's reading of the specification and the GPT-3.5 key-order result, not a measured claim.¹

CRANE, a 2025 paper from the University of Illinois, gives a theoretical reason for this. The authors prove that a model with a fixed number of layers, forced to emit only a string from a very restrictive grammar (for example, only a final yes or no), can solve only problems in a limited circuit class, TC0. Given room for intermediate steps, the same model can do much more.⁶ Their fix is to extend the grammar so that free reasoning comes first, then switch the constraint on between delimiters for the final expression. On GSM-Symbolic with Qwen2.5-Math-7B-Instruct, plain constrained decoding scored 29%, unconstrained chain of thought also 29%, and CRANE 38%. The constrained runs parsed 99% of the time, unconstrained chain of thought 82%, and CRANE 94%.⁶

A subtler problem remains even with a good field order. JSONSchemaBench's authors note that masking only filters tokens but can still push a model off its usual distribution. In their toy example, a model that wants to write "89,000" is blocked from the comma and ends up writing "890000".⁴ Geng et al. found a starker version with beam search. Under their extraction and parsing grammars, the single most likely output was consistently the empty string, because the grammar's structure was unnatural enough to the model that stopping immediately scored higher than any real answer. Length normalization fixed it.³

What the original study could not test

Tam et al. list their own main limitation plainly: "due to cost constraints, we were unable to include results from more powerful language models such as LLaMA 70B or GPT-4o." They say those models might show how the effect changes with model size, and they also call their task set limited in scope.¹ Their strongest model, gpt-4o-mini, was the one where schema enforcement came closest to plain text.¹ The question that started the argument has only been tested on models of that size or smaller, on two-field schemas.