Lesson 2: Parsing Hostile Output

Fences, preambles, and trailing commas

Even with a good schema, raw model output is frequently not parseable JSON. The failures are consistent enough to handle systematically.

import json
import re


def extract_json_text(raw: str) -> str:
    """Pull the JSON payload out of a response that may be wrapped in prose."""
    text = raw.strip()

    fence = re.search(r"```(?:json)?\s*(.*?)```", text, re.DOTALL)
    if fence:
        text = fence.group(1).strip()

    start = text.find("{")
    if start == -1:
        start = text.find("[")
    if start > 0:
        text = text[start:]

    return text.strip()

This handles the markdown fence, with or without a language tag, and a conversational preamble before the JSON begins. It deliberately does not attempt to fix trailing commas or single quotes by rewriting the text, and that restraint is the important part.

Why aggressive repair is a trap. It is tempting to strip trailing commas with a regular expression, convert single quotes to double quotes, and balance brackets. Each of those transformations can silently change meaning. A regular expression removing a comma before a closing brace will also mangle a comma inside a string containing ,}. You end up with valid JSON holding wrong data, which is worse than a parse error, because a parse error is visible.

Prefer this order: extract the likely JSON region with the conservative function above, attempt a strict parse, and if that fails, send the error back to the model rather than rewriting the text yourself. The model repairs its own output far more reliably than a regular expression does.

Where you do want tolerant parsing, use a library built for it rather than hand-rolled regular expressions, and treat its output as a suggestion to be validated rather than a result to be trusted.

Handling max_tokens truncation

Truncation is the failure that looks like a parsing problem and is not.

{"title": "Carbonara", "ingredients": [{"name": "spaghetti", "quan

The output simply stops, because generation hit the token limit. No amount of parsing skill recovers the missing data, because it was never produced.

Detect it from the response metadata, not from the parse failure. Providers report why generation stopped, usually in a field named something like finish_reason or stop_reason. A value indicating the length limit was reached tells you exactly what happened.

def check_completion(response: dict) -> None:
    """Raise a specific error when generation was cut short."""
    reason = response.get("stop_reason") or response.get("finish_reason")
    if reason in {"max_tokens", "length"}:
        raise OutputError("generation truncated at the token limit")

[VOLATILE: field and value names for stop reasons differ between providers. Verify the names for each provider you support.]

Distinguishing truncation from malformed output matters because the responses differ. Malformed output is worth repairing, since the model produced everything and formatted it badly. Truncation is not, since retrying with the same limit produces the same cut. The fixes for truncation are to raise the limit, to reduce what you asked for, or to split the work into smaller pieces.

Three preventive measures. Set the output limit generously for extraction tasks, since output tokens are the expensive half but a truncated response wastes the entire call. Cap the input, because a very long document invites a very long response. And bound your schema with max_length on lists, as Lesson 1 did, so the model has an explicit ceiling rather than discovering one.

The repair loop with an attempt ceiling

When validation fails, you have information the model does not: the specific error. Sending it back is remarkably effective, because the model usually produced nearly correct output and needs to be told what was wrong.

from pydantic import BaseModel, ValidationError

MAX_REPAIR_ATTEMPTS = 2


def extract_with_repair(
    document: Document,
    *,
    schema: type[BaseModel],
) -> BaseModel:
    """Extract structured data, feeding validation errors back for repair."""
    messages = build_extraction_messages(document, schema)

    for attempt in range(MAX_REPAIR_ATTEMPTS + 1):
        response = call_provider(messages)
        check_completion(response)
        raw = response_text(response)

        try:
            return schema.model_validate_json(extract_json_text(raw))
        except (ValidationError, json.JSONDecodeError) as exc:
            if attempt == MAX_REPAIR_ATTEMPTS:
                raise OutputError(
                    f"validation failed after {attempt + 1} attempts"
                ) from exc

            logger.warning(
                "repair attempt %d for %s: %s",
                attempt + 1, document.id, type(exc).__name__,
            )
            messages = messages + [
                {"role": "assistant", "content": raw},
                {"role": "user", "content": repair_instruction(exc)},
            ]

    raise AssertionError("unreachable")


def repair_instruction(exc: Exception) -> str:
    """Turn a validation failure into a correction request."""
    return (
        "Your previous response failed validation with these errors:\n\n"
        f"{exc}\n\n"
        "Return only the corrected JSON object. No explanation, no markdown fence."
    )

Several decisions here are deliberate.

The ceiling is low. Two repairs is usually enough. If a model cannot produce valid output in three attempts, the problem is the schema or the prompt rather than this particular call, and further attempts cost money without changing that.

The full conversation is sent back. The failed response is included as an assistant turn and the correction as a user turn, so the model can see what it produced and what was wrong with it. Sending only the error without the original output performs noticeably worse.

The validation error is passed through directly. Pydantic's error messages name the field, the problem, and the received value, which is exactly the information needed. Do not summarise it into your own wording.

Failure raises a domain error. OutputError comes from the Module 5 hierarchy, so callers already know how to categorise it.

[IMAGE PROMPT M6-1
Purpose: Show the repair loop including the distinction between repairable and non repairable failures, and where the attempt ceiling stops it.
Visual type: Flowchart with a bounded loop.
Prompt: A clean educational flowchart reading top to bottom. Start box labelled "call model". Arrow to a decision diamond labelled "generation truncated?". Its "yes" branch exits right to a terminal box labelled "raise, do not repair" with the sub-label "same limit gives the same cut". Its "no" branch continues down to a box labelled "extract JSON region" then to a decision diamond labelled "validates against schema?". Its "yes" branch exits right to a terminal box labelled "return validated record". Its "no" branch goes down to a decision diamond labelled "attempts remaining?". Its "no" branch exits to a terminal box labelled "raise OutputError". Its "yes" branch goes to a box labelled "append failed output and validation error to messages", from which a return arrow curves up the left side back to the "call model" box, labelled "repair attempt". A bracket beside the loop is labelled "ceiling: 2 repairs".
Required elements: Three decision diamonds with the stated questions, yes and no labels on every branch, three terminal outcomes, the feedback arrow closing the loop, the ceiling bracket.
Style: Clean educational illustration, professional, uncluttered, high contrast, flat vector, diamonds for decisions and rectangles for actions.
Layout: Vertical main flow with terminal outcomes branching right and the repair feedback arrow curving up the left side.
Text labels: "call model", "generation truncated?", "raise, do not repair", "same limit gives the same cut", "extract JSON region", "validates against schema?", "return validated record", "attempts remaining?", "raise OutputError", "append failed output and validation error to messages", "repair attempt", "ceiling: 2 repairs", "yes", "no".
Aspect ratio: 4:3
Accessibility: Label every branch yes or no in text and use distinct shapes for decisions and actions rather than colour.
Avoid: Code screenshots, decorative icons, tiny text, logos, watermarks.
Alt text: Flowchart of a repair loop where truncated generations raise immediately, validation failures append the failed output and error to the conversation and retry, and a two attempt ceiling stops the loop with an OutputError.
END IMAGE PROMPT]

Native structured output modes versus prompt and pray

Providers increasingly offer modes that constrain generation to a schema, so that invalid output is impossible rather than merely discouraged. These go by various names such as structured outputs, JSON mode, or guided decoding. [VOLATILE: names, availability, and exact guarantees differ by provider and change often. Verify before publishing.]

The distinction that matters is what is actually guaranteed.

A strict schema-constrained mode constrains token selection during generation, so the output cannot violate the schema. Fences, preambles, and trailing commas become impossible. This is a strong guarantee and worth using wherever available.

A loose JSON mode guarantees only that the output is syntactically valid JSON, not that it matches your schema. Fields can be missing, extra, or wrongly typed.

Prompting alone guarantees nothing.

Validate regardless of which one you used. This is the part people skip once they enable a strict mode, and it is a mistake for three reasons. Schema conformance is not semantic correctness, which the next section covers. Your validators, coercions, and cross-field rules run in your code and not in the provider's constrained decoder. And a provider can change behaviour, degrade under load, or fall back silently, and your pipeline should not depend on their guarantee holding to remain correct.

The useful framing: a native mode dramatically reduces how often the repair loop runs. It does not remove the need for it.

There is also a cost to know about. Constrained decoding can reduce output quality on tasks requiring reasoning, because the model is being restricted while it generates. For hard extraction, letting the model think in prose first and then produce structure, or splitting into two calls, sometimes beats forcing structure from the first token.

Validating semantics, not just shape

A response can satisfy every schema constraint and still be wrong. Shape validation catches a string where a number belonged. It does not catch a number that is false.

Four semantic checks worth building, using the recipe extractor as the example.

Values that must resolve. If the model returns a source URL or an identifier, check it exists in your data rather than assuming.

if extracted.source_url and extracted.source_url not in document.known_urls:
    raise OutputError("extracted source_url does not appear in the document")

Values that must appear in the input. For extraction specifically, an ingredient not present in the source text was invented.

def check_grounding(extracted: ExtractedRecipe, source: str) -> list[str]:
    """Return the names of ingredients that do not appear in the source text."""
    lowered = source.lower()
    return [
        ing.name for ing in extracted.ingredients
        if ing.name.lower() not in lowered
    ]

This is a blunt check and it produces false positives, since a source saying "eggs" and an extraction saying "egg" is fine. Its value is not as a hard gate but as a signal: a document where six of eight ingredients fail the check is a document to reject or flag, and the rate across a corpus tells you whether extraction quality is drifting.

Dates and quantities that must be possible. A date of February 30th passes a string check and a loose date parse. A cook time of negative ten minutes passes any check that only looks at types.

Internal consistency. The cross-field validator from Lesson 1 checking that total time is not less than its parts is a semantic check, and it belongs in the model definition where it runs automatically.

The general principle: schema validation tells you the answer has the right shape. Semantic validation tells you it might be true. Both are needed, and only the first is free.

Partial results

A batch of ten documents where two fail should produce eight results and two rejections, not zero results.

from dataclasses import dataclass


@dataclass(frozen=True)
class ExtractionOutcome:
    document_id: str
    extracted: ExtractedRecipe | None
    error: str | None

    @property
    def succeeded(self) -> bool:
        return self.extracted is not None


def extract_batch(documents: list[Document]) -> list[ExtractionOutcome]:
    """Extract each document independently. One failure does not lose the others."""
    outcomes: list[ExtractionOutcome] = []
    for document in documents:
        try:
            extracted = extract_with_repair(document, schema=ExtractedRecipe)
            outcomes.append(ExtractionOutcome(document.id, extracted, None))
        except (OutputError, ProviderError) as exc:
            outcomes.append(ExtractionOutcome(document.id, None, str(exc)))
    return outcomes

This is the rejection record pattern from Module 2 applied to model calls. Every input produces an outcome, successes and failures travel together, and the caller decides what to do with each.

Partial results within a single response are the harder case. If you asked for ten extractions in one call and the response contains eight valid records and two malformed ones, validating the whole response as a list means losing all ten.

Validate the items individually instead:

def validate_items(payload: list[dict], schema: type[BaseModel]) -> tuple[list, list]:
    """Validate each item independently, returning valid records and errors."""
    valid, errors = [], []
    for index, item in enumerate(payload):
        try:
            valid.append(schema.model_validate(item))
        except ValidationError as exc:
            errors.append({"index": index, "error": str(exc)})
    return valid, errors

Then repair only the failures, sending back just the items that did not validate, rather than asking for all ten again. This is cheaper and more likely to succeed, because the model is being asked to fix two specific things rather than redo everything.