A chat becomes an API request, and the missing value goes in anyway
Agents act on Web services by filling interface contracts: a JSON Schema, an OpenAPI operation, a function definition. These are real task-oriented dialogues from MultiWOZ and the Schema-Guided Dialogue dataset (built on service APIs), cut just before the user gives a detail the request needs. Each card shows a real model's request body under a required field, and under the same field with one sentence of permission. The endpoint is an illustration; the bodies are the models' outputs, unedited.
Write the permission into the contract, and into the prompt
The same 42 scenarios through native function calling, with the call forced (pooled over models). Only the contract or the system prompt changes.
The user never gave the value. Does the model write one?
Pick a scenario and a model. Each card is that model's real output under one condition, with the field the user never filled marked: red where the model wrote a value, blue where it left the field empty. Verdicts are the paper's strict scoring rule, recomputed here from the released outputs.
All scenarios for this model
Columns: prose, required, nullable, required + license, nullable + license. Click a square to open it.
Permission, not nullability
Click a cell to highlight it in the per-model view. Rows: whether the prompt adds one sentence of permission (the license). Columns: whether the field is required or nullable in the schema.
Languages, real dialogues, native tool calls, other groups' benchmarks
The same comparison, run again where the paper tests generality. Hover a bar for its count.
It is the content of the sentence, not its position or a hint from examples
Where the decision to abstain is made
One per level of access
Every figure in the paper
Rendered from the submitted figure files. Click one to enlarge it with its caption.
Every table in the paper
Everything needed to check the numbers
Rebuild every table and figure (CPU, about a minute)
pip install numpy scipy matplotlib pillow python-dotenv openai python3 scripts/make_tables_figures.py python3 scripts/make_schema_general.py python3 scripts/fig1_arch.py schema python3 scripts/fig_ladders.py schema
Run from the top folder of the archive. The regenerated paper/tables/*.tex match the
shipped copies byte for byte. Re-running an experiment needs API keys (read from environment variables) or a GPU;
see the README inside the archive.