Getting Reliable Structured Output From LLMs: What Actually Works

“Just ask for JSON” works in a demo and breaks in production the first time a model adds a trailing comment, wraps the output in markdown fences, or produces valid JSON that doesn’t match your schema. Reliable structured output isn’t one trick — it’s a stack of defenses, and which layers you need depends on how much failure you can tolerate.

Layer 1: schema-constrained generation, when your provider supports it

The strongest guarantee comes from the model provider actually constraining generation to match a schema at the token level, not just being asked nicely to produce one:

response = client.messages.create(
    model="claude-sonnet-5",
    tools=[{
        "name": "extract_ticket_info",
        "description": "Extract structured info from a support ticket",
        "input_schema": {
            "type": "object",
            "properties": {
                "priority": {"type": "string", "enum": ["low", "medium", "high"]},
                "category": {"type": "string"},
            },
            "required": ["priority", "category"],
        },
    }],
    tool_choice={"type": "tool", "name": "extract_ticket_info"},  # forces this schema
    messages=[{"role": "user", "content": ticket_text}],
)

Forcing a tool call with a defined schema is the most reliable layer available, because the constraint is enforced during generation, not checked after the fact. Use this whenever the provider supports it — it eliminates most of the failure modes below before they can happen.

Layer 2: parse defensively, don’t trust “it said JSON”

Even with schema-constrained output, code that consumes the result should never assume it parses cleanly — providers change behavior, edge cases exist, and a silent parse failure that crashes downstream code is worse than one that’s caught. Every layer below raises a shared exception type, defined once:

class StructuredOutputError(Exception):
    """Raised by any layer below when structured output can't be trusted as-is."""

import json

def parse_structured_response(raw_text: str) -> dict:
    # strip common wrapping artifacts before attempting to parse
    cleaned = raw_text.strip()
    if cleaned.startswith("```"):
        cleaned = cleaned.split("```")[1]
        if cleaned.startswith("json"):
            cleaned = cleaned[4:]
        cleaned = cleaned.rsplit("```", 1)[0]

    try:
        return json.loads(cleaned.strip())
    except json.JSONDecodeError as e:
        raise StructuredOutputError(f"Could not parse response as JSON: {e}\nRaw: {raw_text[:200]}")

The point of surfacing a specific StructuredOutputError instead of letting a raw JSONDecodeError propagate is that it’s now a distinguishable failure mode your eval harness and your logging can specifically track — “how often does structured output actually fail to parse” is a number worth knowing, not something to silently swallow.

Layer 3: validate against the schema, not just “is it valid JSON”

Valid JSON that doesn’t match your expected shape is a distinct failure from invalid JSON, and it’s the one people check for less:

from jsonschema import validate, ValidationError

SCHEMA = {
    "type": "object",
    "properties": {
        "priority": {"type": "string", "enum": ["low", "medium", "high"]},
        "category": {"type": "string"},
    },
    "required": ["priority", "category"],
}

def validate_structured_output(data: dict) -> dict:
    try:
        validate(instance=data, schema=SCHEMA)
        return data
    except ValidationError as e:
        raise StructuredOutputError(f"Output didn't match schema: {e.message}")

This catches the case schema-constrained generation is supposed to prevent but occasionally doesn’t — an enum value slightly off, a field present but the wrong type. Cheap to add, and it turns a downstream KeyError three functions later into an immediate, specific failure at the source.

Layer 4: retry with the error fed back, not from scratch

When validation fails, the cheapest recovery isn’t re-prompting from zero — it’s telling the model specifically what was wrong with its own output and asking it to fix that:

def get_structured_output_with_retry(prompt, schema, max_retries=2):
    messages = [{"role": "user", "content": prompt}]
    for attempt in range(max_retries + 1):
        raw = llm_call(messages)
        try:
            data = parse_structured_response(raw)
            return validate_structured_output(data)
        except StructuredOutputError as e:
            if attempt == max_retries:
                raise
            messages.append({"role": "assistant", "content": raw})
            messages.append({"role": "user", "content": f"That output was invalid: {e}. Fix it and return only valid JSON matching the schema."})

This converges much faster than a fresh retry because the model sees exactly what it got wrong instead of guessing again from the same ambiguous instructions that produced the failure the first time.

Which layers you actually need

Schema-constrained generation (layer 1) when your provider supports it — always, it’s close to free. Defensive parsing and schema validation (layers 2–3) always, regardless of layer 1, because “the provider guarantees it” is a claim worth verifying in your own logs rather than trusting blindly. The retry loop (layer 4) is the one layer worth skipping for latency-sensitive paths where a fast, clear error is better than a slower guaranteed success — not every structured-output use case can afford the extra round trip, and that’s a legitimate tradeoff, not a shortcut.