Structured Output Contracts

Problem and relevance to agents

Valid JSON can contain a false answer. Truncation, refusal and schema violation are also different failures. Collapsing them into one success rate hides the actual behavior.

The project implements a boundary between generation and data consumption: classify each attempt, bound retries and preserve invalid responses for inspection.

Architecture

1. Bounded schemas. Nested objects, discriminated unions and string/array limits in a JSON Schema Draft 2020-12 subset. Strict parsing rejects malformed inputs without silent repair.
2. Explicit policy. Accept, recognize an explicit refusal, retry or exhaust the budget. The experiment permits two attempts per task and uses actual finish metadata.
3. Separate evaluation. Revision-pinned Qwen2-0.5B-Instruct and Qwen2.5-0.5B-Instruct generate locally. Semantic correctness is scored separately, without using gold labels to decide retries.

• Design and methodology

What this demonstrates

• Bounded contracts with explicit refusal and truncation handling.
• Per-schema accounting of attempts, tokens, time and validity separately from correctness.
• Preserved raw failures and policy replay against frozen traces.

Published results

Twelve held-out tasks per model, four per schema family. CPU float32 measurement on Intel Xeon W-3223 with four threads. Qwen2 and Qwen2.5 in the table denote their 0.5B Instruct releases.

Arrows compare initial attempts and final outcomes. Validity and correctness use all tasks as their denominator, including expected abstentions. Task latency includes all attempts and excludes model loading.

Qwen2 ended with 2/12 schema-valid responses and 4/12 correct outcomes, including two correct abstentions. Qwen2.5 reached 6/12 and 6/12 with one correct abstention. Qwen2.5 also fabricated a zero count that passed the schema: structural validation does not verify facts.

The 33 real attempts include 12 refusals, seven malformed responses, four schema violations and two truncations. Fourteen injected fixtures are separate policy controls, not additional model samples. Error analysis.

Reproduce

Replay installs validation dependencies without models. Neural reproduction downloads approximately 1.9 GiB of public snapshots and uses the measured Intel macOS dependency lock.

Run the replay block first to create and activate the environment. Use fresh output directories and one measurement at a time. Original responses are neither repaired nor replaced.

git clone https://github.com/lucioduran/structured-output-contracts.git
cd structured-output-contracts
git checkout ac8680a5b15731932da446694b8a0a5ced00d12a
python3.11 -m venv .venv
. .venv/bin/activate
python -m pip install -r requirements-dev.txt
python -m contracts verify --output results/dev
python -m contracts verify --output results/eval

python -m pip install -r requirements-local.lock
python -m contracts download
OMP_NUM_THREADS=4 MKL_NUM_THREADS=4 OPENBLAS_NUM_THREADS=4 python -m contracts run --split dev --output build/reproduced-dev
OMP_NUM_THREADS=4 MKL_NUM_THREADS=4 OPENBLAS_NUM_THREADS=4 python -m contracts run --split eval --output build/reproduced-eval
python -m contracts verify --output build/reproduced-dev
python -m contracts verify --output build/reproduced-eval
python -m scripts.injected --output build/injected.json

Scope and limitations

• Two small models in one family and twelve tasks per model do not establish production reliability or a general ranking. No independent human review was performed.
• Evaluation covers prompted JSON with validation and retries, not grammar-constrained decoding or hosted structured-output products.
• Exact JSON equality is conservative, not a general factuality metric. Structural bounds do not provide a hard timeout or sandbox.
• One timing observation per task. Paid API cost: USD 0; electricity, hardware and operator time were not measured.

• Function calling and agent output contracts

Sources and revision

Values on this page come from the linked report. They are not live measurements. The report records the executed revision; the reviewed revision includes the subsequent publication of that report.

• Reviewed revision: ac8680a5b15731932da446694b8a0a5ced00d12a
• Revision recorded by the experiment: b73bec6fab382a96691e91a8b39af1491f53da0d
• Report timestamp (UTC): 2026-10-21T22:58:44.050791+00:00
• Release published: v0.1.0 · 2026-10-21T23:02:50Z

• README and usage
• Original evidence
• Results summary
• Original responses and traces
• Development evidence
• Design and methodology
• Error analysis
• Data and provenance
• Frozen configuration
• Models, revisions and licenses
• Measured environment dependencies
• MIT license