Download code and data Real outputs Every figure How to reproduce
The paper's abstract
The whole study in plain words
An everyday pictureThink of a paper form with a box marked required. Someone filling it in who does not know the answer will often write something plausible rather than leave the box empty, unless the form says “leave it blank if you don’t know”. Marking the box optional helps a little. Saying in words that a blank is fine helps much more. The paper finds the same in language models: a required slot reads as “fill me”, and models need to be told, not shown, that empty is acceptable.
New to the words? The glossary explains every term on this page in one or two sentences.
The paper in 3D: from a chat to a wrong API call, and how one sentence stops it
How to use itThis is the agent stack of the paper’s Figure 1, built as a street of buildings. On the left, the user’s chat. Next, the agent’s prompt and its output contract (the JSON form). In the middle, the language model as a tower of layers. On the right, the JSON it writes, the format check, and the Web service that acts on it. The ten steps follow the paper: the setting, the failure, its three questions (Q1 what causes it? Q2 how general is it? Q3 where in the model?), the three fixes and the conclusion. The caption above the picture says what each step shows and where to look, and the colours button explains the colours. Press play or click the numbers; drag to turn the scene, use + and − to zoom, and hover any part for its name and number.
How much we tested
A claim that one sentence works “across models” is only as strong as the range of models and tests behind it. Here is the size of the study, as the paper reports it.
The test sets: where the missing detail comes from
The conditions: what changes between runs
Every condition runs on the same scenarios, so each comparison is paired: the same model, the same chat, one change.
One failure, three questions
In plain wordsAn agent fills a JSON form for a Web service. The user never gave one of the details. Does the model leave that field empty, or make a value up? The paper asks what causes the made-up value, how widely the answer holds, and where inside the model the choice is made. Then it fixes it at three levels of access.
The user never gave the value. Does the model write one?
In plain wordsPick one of the 42 short chats and one model. You see the model’s real answer five times: once in prose and four times as a JSON record (required or nullable field, without or with the permission sentence). Red marks a made-up value, blue a field left empty. The grid below shows the same for every chat at once.
Each card is that model's real output under one condition, with the field the user never filled marked: red where the model wrote a value, blue where it left the field empty. Verdicts follow the paper's strict scoring rule.
All scenarios for this model
Columns: prose, required, nullable, required + license, nullable + license. Click a square to open it.
Required JSON makes models invent values
In plain wordsThe same request, asked two ways. In prose, models usually say they don’t know the detail or leave a blank to fill in. In a JSON record with a required field, they usually write a value they were never given. Listing the same fields in prose already raises the rate: an empty, named slot pulls a value into it.
Permission, not nullability
In plain wordsTwo things could stop the invention: changing the field’s type so that null is allowed (nullable), or adding one sentence that permits null (the license). We tried all four combinations. The type alone helps only partly. The sentence brings invention down whatever the type, and once it is there, the type no longer matters.
Rows: whether the prompt adds the license. Columns: whether the field is required or nullable. The key comparison is the diagonal: a required field with the license against a nullable field without it.
Is it the content of the license, or its position?
In plain wordsMaybe the sentence works only because it is a late, explicit instruction. A sham sentence of the same length and place, with no permission in it, does not help. The real one works at the start of the prompt too. So it is what the sentence says that matters.
Across models, languages, real dialogues, tool calls and benchmarks built by others
In plain wordsThe same comparison, repeated where real agents work: in Spanish, Chinese and Bengali; on real conversations collected by others; through the function-calling feature of the APIs; and on two test sets built by other research groups. The sentence helps everywhere. Where the answer has to be inferred rather than stated, it helps but is not enough alone, and an explicit escape value does better.
Robustness checks
A mid-to-late band of layers makes the decision
In plain wordsWe run an open model twice: once where it leaves the field empty, once where it makes a value up. Then we copy the model’s internal state at one layer from the first run into the second. If that alone makes it write null, that layer carries the decision. Early layers never do; a band in the later middle of the model does. There, the permission sentence restores the empty field more than the nullable type does.
One fix per level of access
In plain wordsHow you can fix it depends on what you control. With only an API, add the sentence (fix 1). If you see the model’s scores for each next word, decode with the sentence’s effect pushed a little further (fix 2, LCD). If you have the weights, a small probe reads the model’s internal state, predicts that the detail was never given, and writes null in that field (fix 3, GCA), in one pass.
Why it happens: a completion habit that permission overrides
In plain wordsModels have seen far more complete records than records with empty fields, so a required slot reads as “fill me”. Showing them examples with empty fields barely changes this. Telling them, in one sentence, that empty is fine does. Models need to be told, not shown.
What this means for Web API contracts
In plain wordsBelow is a real conversation between a person and a booking service. It stops just before the person gives one detail. On the right is what a real model sent to the service: with a required field it made the detail up; with one added sentence of permission it left the field empty. Further down, the same test with native tool calls.
A real dialogue becomes an API request, and the missing value goes in anyway
Agents act on Web services by filling interface contracts: a JSON Schema, an OpenAPI operation, a function definition. These are real task-oriented dialogues from MultiWOZ and the Schema-Guided Dialogue dataset (built on service APIs), cut just before the user gives a detail the request needs. Each card shows a real model's request body under a required field, and under the same field with one sentence of permission. The endpoint is an illustration; the bodies are the models' outputs, unedited.
Write the permission into the contract, and into the prompt
The same 42 scenarios through native function calling, with the call forced (pooled over models). Only the contract or the system prompt changes.
Glossary: the terms on this page, in plain words
Every table in the paper and its supplement
In plain wordsAll tables of the paper and its supplement, each with the paper’s caption. Type in the box to find one; click a row to open it.
Every figure in the paper and its supplement
In plain wordsAll figures, with the paper’s captions. Click one to enlarge it.
All the code, as used
In plain wordsThese are the scripts that ran the experiments and the scripts that draw every table and figure of the paper. Pick a file to read it here.
Every model output and every result file
In plain wordsEvery number in the paper comes from one of these files: the raw record of every model call, the scored results, and the third-party benchmark files with their licences.
Rebuild the paper's tables and figures in a minute
In plain wordsDownload one archive and run a few commands on an ordinary computer to rebuild every table and figure. Re-running the experiments themselves needs API keys, and a graphics card (GPU) for the open models.
Download the archive, unzip it, and run these commands from its top folder.
pip install numpy scipy matplotlib pillow python-dotenv openai python3 scripts/make_tables_figures.py python3 scripts/make_schema_general.py python3 scripts/fig1_arch.py schema python3 scripts/fig_ladders.py schema
Where the method applies
In plain wordsWhat our results do not show, in the paper’s own words.