Lesson 4: Records

@dataclass as a record type

You have been passing dictionaries around since Lesson 1. This lesson replaces them, and it does so before covering object-oriented programming, deliberately. A record is not an object in the full sense. It is a named bundle of fields, and you can get most of the benefit of typed code without inheritance, methods, or any of the design questions that come with classes.

The problem with dictionaries as records:

doc = {"path": "recipes/carbonara.html", "content": "...", "hash": "a3f8..."}

print(doc["contnet"])       # KeyError, at runtime, wherever this line is
print(doc["title"])         # KeyError, this field never existed
doc["hash"] = 42            # accepted silently, now the type is wrong

Nothing here is caught before running. Your editor cannot autocomplete the keys, mypy cannot check them, and a typo six functions away from where the dict was created produces an error that names the typo but not the origin.

A dataclass fixes this with a decorator that generates the boilerplate for you.

from dataclasses import dataclass


@dataclass
class Document:
    path: str
    content: str
    content_hash: str

That is the whole definition. The @dataclass decorator reads the annotated fields and generates an __init__ that accepts them, a __repr__ that prints them readably, and an __eq__ that compares two documents field by field.

doc = Document(path="recipes/carbonara.html", content="...", content_hash="a3f8...")

doc.content_hash          # autocompletes in your editor
doc.contnet               # mypy: "Document" has no attribute "contnet"
print(doc)                # Document(path='recipes/carbonara.html', content='...', ...)

The typo is now caught by mypy before the code runs, which is the point of the strict setting from Module 1. Printing gives you something readable rather than a memory address. Two documents with identical fields compare as equal, which makes tests straightforward.

[IMAGE PROMPT M2-4
Purpose: Compare a dictionary and a dataclass on when errors are caught and what tooling can see.
Visual type: Two-column comparison table rendered as a diagram with a timeline element.
Prompt: A clean educational comparison diagram with two columns headed "dict" and "@dataclass". Four rows run across both columns, each labelled on the far left: "typo in field name", "missing field", "wrong type assigned", "editor autocomplete". In the dict column each of the first three cells contains a cross marker with the text "found at runtime" and the fourth contains a cross marker with "not available". In the dataclass column the first three cells contain a check marker with the text "found before running" and the fourth contains a check marker with "available". Beneath the table runs a short horizontal timeline with two marked points labelled "write code" and "run code", and two arrows pointing to it: one from the dict column landing on "run code" and one from the dataclass column landing on "write code".
Required elements: Two labelled columns, four labelled rows, check and cross markers with accompanying text in every cell, a timeline beneath with two labelled points and two arrows.
Style: Clean educational illustration, professional, uncluttered, high contrast, flat vector, clear table gridlines.
Layout: Table occupying the upper two thirds, timeline centred beneath it.
Text labels: "dict", "@dataclass", "typo in field name", "missing field", "wrong type assigned", "editor autocomplete", "found at runtime", "found before running", "not available", "available", "write code", "run code".
Aspect ratio: 16:9
Accessibility: Pair every check and cross marker with explicit text so the meaning does not depend on the symbol or on colour.
Avoid: Code screenshots, decorative icons, tiny text, logos, watermarks.
Alt text: Comparison table showing that dictionaries surface field typos, missing fields, and wrong types only at runtime with no editor autocomplete, while dataclasses surface all three before running and provide autocomplete.
END IMAGE PROMPT]

field(), defaults, and frozen=True

Defaults work as they do in a function signature, and fields with defaults must come after fields without them.

@dataclass
class Document:
    path: str
    content: str
    content_hash: str
    status: str = "pending"
    word_count: int = 0

Mutable defaults need field(). Module 1 covered why a mutable default argument is shared between calls. The same hazard exists here, and dataclasses refuse to let you make the mistake:

from dataclasses import dataclass, field


@dataclass
class Document:
    path: str
    content: str
    tags: list[str] = []                          # ValueError at class definition
    tags: list[str] = field(default_factory=list) # correct

default_factory takes a callable that is called once per instance, producing a fresh list for every document. Python raises an error for the first form rather than silently sharing, which is a rare case of the language protecting you from this class of bug.

field() does more than defaults:

@dataclass
class Document:
    path: str
    content: str = field(repr=False)              # keep it out of printed output
    tags: list[str] = field(default_factory=list)
    parsed_at: datetime = field(default_factory=datetime.now)
    internal_score: float = field(default=0.0, compare=False)  # ignore in equality

repr=False is genuinely useful for a content field holding 400,000 characters, since otherwise every log line and every debugger view dumps the whole document.

frozen=True and immutability. A frozen dataclass cannot be modified after creation.

@dataclass(frozen=True)
class Document:
    path: str
    content: str
    content_hash: str


doc = Document(path="a.html", content="...", content_hash="a3f8")
doc.content = "something else"     # FrozenInstanceError

This sounds restrictive and is genuinely valuable in a pipeline, for three reasons.

First, it eliminates action at a distance. If a document is frozen, no function further down the pipeline can quietly alter it, so when a value is wrong you only have to look at where it was created rather than everywhere it has been.

Second, frozen dataclasses are hashable, so they can go into sets and be used as dictionary keys. Mutable ones cannot, for the same reason lists cannot.

Third, it makes transformation explicit. Instead of mutating, you produce a new record:

from dataclasses import replace

cleaned = replace(doc, content=normalize(doc.content), status="cleaned")

replace() copies the record with the named fields changed. The original still exists, so you can compare before and after, and a bug in normalize cannot corrupt data that other code is holding.

The cost is that creating new objects instead of mutating uses more memory and more allocations. In a pipeline processing millions of records, that occasionally matters. Start frozen and unfreeze deliberately when you have a measured reason, rather than the reverse.

Document(id=...) over doc["content"]

The practical difference is worth seeing at the call site. Here is Lesson 1's function returning dictionaries, then the same function returning records.

# Before
documents.append({"path": str(path), "content": normalized, "hash": content_hash})

# Somewhere else, much later
for doc in documents:
    if doc["hash"] in index:            # KeyError if the key was named content_hash
        ...

# After
documents.append(
    Document(path=str(path), content=normalized, content_hash=content_hash)
)

# Somewhere else, much later
for doc in documents:
    if doc.content_hash in index:       # mypy catches a wrong name here
        ...

The dictionary version has one field name defined in one place and used in another, with nothing connecting them. Rename the key at creation and every reader breaks silently until it runs. The record version has a single definition that both sides refer to, so a rename is caught immediately everywhere.

There is a second, subtler benefit. Document(...) is self-documenting at the point of creation. The keyword arguments name every value, so a reader does not need to look elsewhere to know what the third string means.

The general principle: give data a type as early as possible and keep it typed. Lesson 5 extends this to data arriving from outside your program.

Status and provenance fields

A record should carry enough information to answer questions about itself later, without anyone needing to reconstruct where it came from.

Provenance means the origin and history of a piece of data. In practice it is a small set of fields recording where the record came from and what has happened to it.

from dataclasses import dataclass, field
from datetime import datetime, timezone


@dataclass(frozen=True)
class Document:
    # Identity
    id: str
    content_hash: str

    # Payload
    content: str = field(repr=False)

    # Provenance
    source_path: str
    source_url: str | None = None
    ingested_at: datetime = field(
        default_factory=lambda: datetime.now(timezone.utc)
    )

    # Status
    status: str = "pending"
    rejection_reason: str | None = None

Why each group earns its place:

Identity. A stable id lets you refer to this document from a log line, a database row, or an error report without carrying the whole object. The content_hash serves deduplication and also tells you whether a re-ingested document actually changed.

Provenance. Six months from now, someone will ask why a particular answer cited a particular fact. Without source_path and source_url, that question is unanswerable. With them it takes seconds. Recording ingested_at in UTC with an explicit timezone matters, because a naive datetime is ambiguous the moment your code runs on a machine in a different zone.

Status. Carrying status on the record means a rejected document can travel through the pipeline alongside accepted ones rather than vanishing. That is what makes a rejection report possible: you have the object, its reason, and its origin all in one place.

Here is the corpus function from Lesson 1, rewritten with records, which is what it should have been all along.

import hashlib
from collections import Counter
from pathlib import Path

MIN_CONTENT_LENGTH = 50


def prepare_corpus(folder: Path) -> tuple[list[Document], list[Document]]:
    """Return accepted documents and rejected documents, each with a reason."""
    seen_hashes: set[str] = set()
    accepted: list[Document] = []
    rejected: list[Document] = []

    for path in sorted(folder.iterdir()):
        if not path.is_file():
            continue

        try:
            content = path.read_text(encoding="utf-8")
        except (UnicodeDecodeError, OSError):
            rejected.append(_rejected(path, "", "not_readable"))
            continue

        normalized = content.strip()
        content_hash = hashlib.sha256(normalized.encode("utf-8")).hexdigest()

        if len(normalized) < MIN_CONTENT_LENGTH:
            rejected.append(_rejected(path, content_hash, "too_short"))
            continue

        if content_hash in seen_hashes:
            rejected.append(_rejected(path, content_hash, "duplicate"))
            continue

        seen_hashes.add(content_hash)
        accepted.append(
            Document(
                id=content_hash[:16],
                content_hash=content_hash,
                content=normalized,
                source_path=str(path),
                status="valid",
            )
        )

    return accepted, rejected


def _rejected(path: Path, content_hash: str, reason: str) -> Document:
    return Document(
        id=content_hash[:16] or path.name,
        content_hash=content_hash,
        content="",
        source_path=str(path),
        status="rejected",
        rejection_reason=reason,
    )

Compare the return type to the original. tuple[list[dict], Counter] told you almost nothing. tuple[list[Document], list[Document]] tells you what you are getting, and the rejection report is now derivable with one line:

Counter(doc.rejection_reason for doc in rejected)

The counts were never the real output. The rejected documents were, and the counts fall out of them.