THE OUTPUT CONTRACT CARD

OUTPUT CONTRACT CHECK · outputs.jsonl against contract.json · 48 outputs
PASS 38 · FAIL 10 · failure rate 20.8% (95% range 11.7% to 34.3%)

BY CAUSE · an output can carry more than one cause
    3 × code fence · lines 4, 18, 32
        fix: constrain the reply instead of asking for JSON in the prompt: structured outputs (output_config.format with type json_schema) on a model that supports them, or strict: true on the tool whose input carries the data. Stopgap: strip the fence before parsing; the reply is still unconstrained.
    2 × enum value differs only in case at /currency · lines 13, 41
        fix: the docs say Claude may return an enum value that differs from the schema only in capitalization: compare case-insensitively, or normalise the case before the downstream check.
    1 × cut off at max_tokens · lines 23
        fix: the reply hit max_tokens: raise max_tokens (the docs: retry with a higher max_tokens to get the complete structured output) and read stop_reason before parsing.
    1 × extra field at /notes · lines 45
        fix: set additionalProperties: false on every object in the schema you send (structured outputs require false for objects), so extra keys cannot appear.
    1 × missing required field at /issue_date · lines 36
        fix: send this contract as the schema (structured outputs, or strict tool use): required fields, types, enums and additionalProperties false are then enforced. If you already do, the schema you send is looser than the contract: run tighten.
    1 × prose around the JSON · lines 10
        fix: constrain the reply instead of asking for JSON in the prompt: structured outputs (output_config.format with type json_schema) on a model that supports them, or strict: true on the tool whose input carries the data. Stopgap: take the first JSON value with a decoder; the reply is still unconstrained.
    1 × wrong type at /total · lines 28
        fix: send this contract as the schema (structured outputs, or strict tool use), so each field can only take its type. If you already do, the schema you send is looser than the contract: run tighten.

EVERY FAILURE
  line 4 · out-03 · stop_reason end_turn
      code fence (the JSON inside passes the contract)
  line 10 · out-09 · stop_reason end_turn
      prose around the JSON (the JSON inside passes the contract)
[… 21 more lines, trimmed here]

PLATFORM FIT · contract.json under structured outputs and strict tool use · rules as read 2026-10-12

ENFORCED BY THE PLATFORM (constrained decoding)
  /                            type object
  /                            required invoice_number, supplier, issue_date, currency, total, lines
  /                            additionalProperties false
  /properties/invoice_number   type string
  /properties/invoice_number   pattern ^INV-[0-9]{5}$ (simple patterns work; complex ones may be refused)
  /properties/supplier         type string
  /properties/issue_date       type string
  /properties/issue_date       format date
  /properties/currency         type string
  /properties/currency         enum ["USD", "EUR", "GBP"]
  /properties/total            type number
  /properties/lines            type array
  /properties/lines            minItems 1
  /properties/lines/items      type object
  /properties/lines/items      required description, quantity, unit_price
  /properties/lines/items      additionalProperties false
  /properties/lines/items/properties/description type string
  /properties/lines/items/properties/quantity type integer
  /properties/lines/items/properties/unit_price type number

NOT SUPPORTED · strip it from the schema you send, say it in the field's description, keep checking it after the call (this script's check)
  /properties/supplier         minLength 1
  /properties/total            minimum 0
  /properties/lines/items/properties/quantity minimum 1

REFUSED · sent as is, the request is refused (400); change it
  none
[… 16 more lines, trimmed here]

PASS a clean output, a tool_use input and text blocks all pass
PASS a fenced reply is caught as a code fence and its inside still checked
PASS prose around the JSON is caught
PASS a reply cut off at max_tokens is named from stop_reason
PASS a refusal is named from stop_reason, not called a format fault
PASS an enum value not listed and one that differs only in case are told apart
PASS a wrong type, a missing field and an extra key are each named with their path
PASS pattern, minimum and format failures are all reported on one output
PASS a raw line that is not JSON is checked as text
PASS the failure rate counts outputs, not causes: 11 of 14
PASS the Wilson range for 8 of 10 is 49.0% to 94.3%
PASS the Wilson range for 0 of 40 starts at 0 and stays above 0 (8.8%)
PASS each cause names its fix: fences to structured outputs, case to normalising, max_tokens to raising it, refusal to stop_reason, bounds to checking after
PASS --field picks the field that holds the output
PASS platform: minimum is not supported (strip and check after); the enum and pattern are enforced; FITS is not claimed
PASS platform: additionalProperties true and a recursive schema are refused; oneOf is flagged as unlisted
PASS platform: 25 optional parameters exceed the limit of 24 and are refused
PASS platform: a contract inside the limits fits as written
PASS tighten: a dropped required field, an enum left open, a widened type and extra keys allowed are all found
PASS the validator: true is not 1, 1.0 is an integer, $defs refs resolve, oneOf counts matches
PASS the same outputs give the same report, byte for byte
PASS the command line: all-pass exits 0, failures exit 1, --max-rate sets the bar, --json writes, --help shows usage
PASS assets/api-rules.json matches the rules built into this machine
PASS nothing in this machine opens a network connection
selftest 24/24