The whole study in plain words

An everyday pictureThink of a paper form with a box marked required. Someone filling it in who does not know the answer will often write something plausible rather than leave the box empty, unless the form says “leave it blank if you don’t know”. Marking the box optional helps a little. Saying in words that a blank is fine helps much more. The paper finds the same in language models: a required slot reads as “fill me”, and models need to be told, not shown, that empty is acceptable.

New to the words? The glossary explains every term on this page in one or two sentences.

How much we tested

One failure, three questions

In plain wordsAn agent fills a JSON form for a Web service. The user never gave one of the details. Does the model leave that field empty, or make a value up? The paper asks what causes the made-up value, how widely the answer holds, and where inside the model the choice is made. Then it fixes it at three levels of access.

Figure 1 of the paper · where schema slot pressure arises in an agent stack, and how the paper tests it

The user never gave the value. Does the model write one?

In plain wordsPick one of the 42 short chats and one model. You see the model’s real answer five times: once in prose and four times as a JSON record (required or nullable field, without or with the permission sentence). Red marks a made-up value, blue a field left empty. The grid below shows the same for every chat at once.

Each card is that model's real output under one condition, with the field the user never filled marked: red where the model wrote a value, blue where it left the field empty. Verdicts follow the paper's strict scoring rule.

Required JSON makes models invent values

In plain wordsThe same request, asked two ways. In prose, models usually say they don’t know the detail or leave a blank to fill in. In a JSON record with a required field, they usually write a value they were never given. Listing the same fields in prose already raises the rate: an empty, named slot pulls a value into it.

Permission, not nullability

In plain wordsTwo things could stop the invention: changing the field’s type so that null is allowed (nullable), or adding one sentence that permits null (the license). We tried all four combinations. The type alone helps only partly. The sentence brings invention down whatever the type, and once it is there, the type no longer matters.

Is it the content of the license, or its position?

In plain wordsMaybe the sentence works only because it is a late, explicit instruction. A sham sentence of the same length and place, with no permission in it, does not help. The real one works at the start of the prompt too. So it is what the sentence says that matters.

Across models, languages, real dialogues, tool calls and benchmarks built by others

In plain wordsThe same comparison, repeated where real agents work: in Spanish, Chinese and Bengali; on real conversations collected by others; through the function-calling feature of the APIs; and on two test sets built by other research groups. The sentence helps everywhere. Where the answer has to be inferred rather than stated, it helps but is not enough alone, and an explicit escape value does better.

Robustness checks

A mid-to-late band of layers makes the decision

In plain wordsWe run an open model twice: once where it leaves the field empty, once where it makes a value up. Then we copy the model’s internal state at one layer from the first run into the second. If that alone makes it write null, that layer carries the decision. Early layers never do; a band in the later middle of the model does. There, the permission sentence restores the empty field more than the nullable type does.

One fix per level of access

In plain wordsHow you can fix it depends on what you control. With only an API, add the sentence (fix 1). If you see the model’s scores for each next word, decode with the sentence’s effect pushed a little further (fix 2, LCD). If you have the weights, a small probe reads the model’s internal state, predicts that the detail was never given, and writes null in that field (fix 3, GCA), in one pass.

Figure 2 of the paper · three fixes, one per level of access

Why it happens: a completion habit that permission overrides

In plain wordsModels have seen far more complete records than records with empty fields, so a required slot reads as “fill me”. Showing them examples with empty fields barely changes this. Telling them, in one sentence, that empty is fine does. Models need to be told, not shown.

What this means for Web API contracts

In plain wordsBelow is a real conversation between a person and a booking service. It stops just before the person gives one detail. On the right is what a real model sent to the service: with a required field it made the detail up; with one added sentence of permission it left the field empty. Further down, the same test with native tool calls.

A real dialogue becomes an API request, and the missing value goes in anyway

Agents act on Web services by filling interface contracts: a JSON Schema, an OpenAPI operation, a function definition. These are real task-oriented dialogues from MultiWOZ and the Schema-Guided Dialogue dataset (built on service APIs), cut just before the user gives a detail the request needs. Each card shows a real model's request body under a required field, and under the same field with one sentence of permission. The endpoint is an illustration; the bodies are the models' outputs, unedited.

Write the permission into the contract, and into the prompt

The same 42 scenarios through native function calling, with the call forced (pooled over models). Only the contract or the system prompt changes.

Glossary: the terms on this page, in plain words

Every table in the paper and its supplement

In plain wordsAll tables of the paper and its supplement, each with the paper’s caption. Type in the box to find one; click a row to open it.

Every figure in the paper and its supplement

In plain wordsAll figures, with the paper’s captions. Click one to enlarge it.

All the code, as used

In plain wordsThese are the scripts that ran the experiments and the scripts that draw every table and figure of the paper. Pick a file to read it here.

Every model output and every result file

In plain wordsEvery number in the paper comes from one of these files: the raw record of every model call, the scored results, and the third-party benchmark files with their licences.

Rebuild the paper's tables and figures in a minute

In plain wordsDownload one archive and run a few commands on an ordinary computer to rebuild every table and figure. Re-running the experiments themselves needs API keys, and a graphics card (GPU) for the open models.

pip install numpy scipy matplotlib pillow python-dotenv openai
python3 scripts/make_tables_figures.py
python3 scripts/make_schema_general.py
python3 scripts/fig1_arch.py schema
python3 scripts/fig_ladders.py schema