Lesson 1: Files and Formats

The problem this lesson solves

The recipe-extractor project has worked on a test folder of two hundred scraped pages since Module 2. Someone now hands you the real corpus: two million files, forty gigabytes, collected over three years by four different scrapers.

The first run fails in a way that has nothing to do with your logic. A file encoded in Windows-1252 rather than UTF-8 raises UnicodeDecodeError and kills the job at document 4,000. You fix that, and the next run dies at document 190,000 because a path contains a character your string manipulation mangled. You fix that, and the third run consumes 31 gigabytes of memory and the machine stops responding.

None of these are exotic. They are the standard failures of moving from a folder you created to data someone else created, and this module is about surviving them.

pathlib over string paths

Paths are not strings. Treating them as strings works until it does not, and the failures are platform specific, which means they appear on a colleague's machine or in production rather than yours.

# String paths
import os

path = folder + "/" + filename                    # breaks on Windows
path = os.path.join(folder, filename)             # better, but verbose
name = filename.split(".")[0]                     # breaks on "recipe.v2.html"
ext = filename[filename.rfind("."):]              # breaks when there is no dot
parent = os.path.dirname(os.path.abspath(path))

pathlib replaces all of it with objects that know what a path is.

from pathlib import Path

folder = Path("data/scraped")
path = folder / "recipe.html"                     # the / operator joins correctly

path.name          # 'recipe.html'
path.stem          # 'recipe'         and 'recipe.v2' for recipe.v2.html
path.suffix        # '.html'          and '' when there is no extension
path.parent        # Path('data/scraped')
path.exists()      # True or False
path.is_file()     # True for files, False for directories
path.stat().st_size  # size in bytes

The / operator is the piece people find odd and then never give up. It joins with the correct separator for the platform, so the same code runs on Linux, macOS, and Windows.

Reading and writing.

content = path.read_text(encoding="utf-8")
path.write_text(content, encoding="utf-8")

raw = path.read_bytes()
path.write_bytes(raw)

Note encoding="utf-8" written explicitly. Without it, Python uses a platform-dependent default, which means the same file read on two machines can produce different results. Always state the encoding.

Walking a directory.

folder.iterdir()                    # immediate children, files and directories
folder.glob("*.html")               # matching files in this folder
folder.rglob("*.html")              # matching files at any depth
folder.rglob("*.[hj]*")             # simple wildcards are supported

All of these return generators rather than lists, which matters at two million files. list(folder.iterdir()) on a large directory builds the whole listing in memory before you touch a single file. Iterating directly starts work immediately.

Creating and removing.

output = Path("data/processed")
output.mkdir(parents=True, exist_ok=True)     # create the whole path, no error if present

exist_ok=True is the difference between a job that can be rerun and one that fails on its second invocation. parents=True creates intermediate directories rather than requiring them to exist.

One caution. A Path is not a string, so passing one to an older library that expects a string sometimes fails. Convert explicitly with str(path) at that boundary, and keep Path objects everywhere else. The Document record from Module 2 stores source_path as a string for exactly this reason: it is a value being recorded, not a path being used.

Text encodings, and the errors that appear only on someone else's data

An encoding is the mapping between the bytes in a file and the characters they represent. UTF-8 is the near-universal standard now, but "now" does not describe a corpus assembled over three years from thousands of websites.

content = path.read_text(encoding="utf-8")
# UnicodeDecodeError: 'utf-8' codec can't decode byte 0x92 in position 1247

That byte is a curly apostrophe in Windows-1252, which is not valid UTF-8. The file is not corrupt. It is simply a different encoding, and your program guessed wrong.

Four strategies, and when each is right.

# 1. Strict, the default. Raises on invalid bytes.
path.read_text(encoding="utf-8")

# 2. Replace. Invalid bytes become the replacement character.
path.read_text(encoding="utf-8", errors="replace")

# 3. Ignore. Invalid bytes are dropped silently.
path.read_text(encoding="utf-8", errors="ignore")

# 4. Detect, then decode.
raw = path.read_bytes()
detected = detect_encoding(raw)
text = raw.decode(detected, errors="replace")

Strict is right when you control the data and any bad file is a real problem you want to hear about. Replace is right for a mixed corpus, because you keep the document and can see where the damage is. Ignore is almost never right, since it destroys information silently and you cannot tell later whether a document was clean.

Detection is right when the corpus genuinely contains several encodings and you care about accuracy. Libraries such as charset-normalizer and chardet guess from byte patterns. They are good and not perfect, so treat the result as a best guess rather than a fact, and record which encoding was used on the document record.

The practical recommendation for a scraped corpus:

def read_document_text(path: Path) -> tuple[str, str] | None:
    """Return (text, encoding_used), or None if the file is not text at all."""
    raw = path.read_bytes()

    if b"\x00" in raw[:8192]:          # null bytes strongly suggest binary
        return None

    for encoding in ("utf-8", "windows-1252", "latin-1"):
        try:
            return raw.decode(encoding), encoding
        except UnicodeDecodeError:
            continue

    return raw.decode("utf-8", errors="replace"), "utf-8-replaced"

Try the likely encodings in order, fall back to lossy decoding, and always report which path was taken so the document record carries it. Note that latin-1 never raises, because every byte sequence is valid in it, which is why it goes last and why "it decoded fine" does not mean "it decoded correctly".

Unicode normalization. Two strings that look identical can differ in bytes. The character "é" can be one code point or two, an "e" followed by a combining accent. They render the same and compare as unequal, which breaks deduplication.

import unicodedata

normalized = unicodedata.normalize("NFC", text)

Normalizing to NFC before hashing fixes the deduplication bug flagged in Module 2's edge case list. This is the sort of defect that is invisible in testing and obvious in a corpus containing text from several sources.

A related trap: the BOM. Some files begin with a byte order mark, which decodes as an invisible character at the start of your text. Reading with encoding="utf-8-sig" strips it. A stray BOM at the start of a JSON file causes a parse error that looks inexplicable, because the offending character does not appear when you print the line.

JSON and JSONL

JSON is the interchange format you will meet everywhere. Python's json module maps it to built-in types.

import json

data = json.loads(text)                 # from a string
data = json.load(file_object)           # from an open file

text = json.dumps(data)                 # to a string
text = json.dumps(data, ensure_ascii=False, indent=2)

Two arguments worth knowing. ensure_ascii=False keeps non-English characters readable instead of escaping them as \uXXXX, which matters for a multilingual corpus. indent=2 produces human-readable output, and should be omitted for machine-consumed files because it inflates size for no benefit.

Safe extraction from nested structures. Module 2 covered accessing nested payloads by shape rather than position. The same care applies to JSON loaded from disk, since a file written by someone else may not have the structure you expect.

def extract_recipe_fields(payload: dict) -> dict:
    """Pull the fields we care about, tolerating a missing or unexpected shape."""
    recipe = payload.get("recipe", {})
    if not isinstance(recipe, dict):
        return {}

    ingredients = recipe.get("ingredients", [])
    if not isinstance(ingredients, list):
        ingredients = []

    return {
        "title": recipe.get("name", ""),
        "ingredients": [str(item) for item in ingredients],
        "servings": recipe.get("recipeYield"),
    }

The isinstance checks are not paranoia. Scraped JSON regularly contains a string where a list was expected, because a website changed its markup. This is defensive code at a boundary, which Module 2 established is exactly where defensive code belongs, and note that it is a single function rather than checks scattered through the pipeline.

JSONL, meaning JSON Lines, is one JSON object per line with no wrapping array.

{"id": "a3f8", "title": "Carbonara", "servings": 4}
{"id": "b2c1", "title": "Cacio e Pepe", "servings": 2}
{"id": "c9d4", "title": "Amatriciana", "servings": 4}

Reading and writing are both trivial:

def read_jsonl(path: Path) -> Iterator[dict]:
    with path.open(encoding="utf-8") as f:
        for line in f:
            line = line.strip()
            if line:
                yield json.loads(line)


def write_jsonl(path: Path, records: Iterable[dict]) -> None:
    with path.open("w", encoding="utf-8") as f:
        for record in records:
            f.write(json.dumps(record, ensure_ascii=False) + "\n")

Why JSONL is the default interchange format

The difference between JSON and JSONL looks cosmetic and is decisive at scale.

A JSON array must be read whole. The structure is not valid until the closing bracket arrives, so a parser cannot hand you the first object until it has consumed the entire file. A four gigabyte JSON array requires four gigabytes of file plus the memory for the parsed objects, all at once, before you can process record one.

JSONL is readable one line at a time. Each line is independently valid, so you can process record one while record two million is still on disk. Memory stays flat regardless of file size.

That single property produces the rest of the advantages.

Appending is trivial. Adding a record to JSONL means writing one more line. Adding to a JSON array means rewriting the file, or seeking to before the closing bracket and hoping nothing else is writing.

Partial files remain usable. If a job is killed mid-write, a JSONL file is valid up to the last complete line, and every record before the failure is readable. A truncated JSON array is entirely unparseable, so an interrupted job loses everything.

A corrupt record costs one record. A malformed line can be caught, logged, and skipped. A malformed JSON array fails as a whole.

Standard text tools work. wc -l counts records, head samples them, split divides a file into chunks, and grep finds records containing a term. None of that works on a pretty-printed array.

[IMAGE PROMPT M4-1
Purpose: Show why a JSON array must be parsed as a whole while JSONL can be read one record at a time, and what each does when a write is interrupted.
Visual type: Two-panel structural comparison with a failure case.
Prompt: A clean educational side-by-side comparison. The left panel is headed "JSON array" and shows a tall file block with an opening bracket at the top, several record rows in the middle, and a closing bracket at the bottom. A bracket spanning the entire block is labelled "must read all of it before any record is usable". Beneath it, a second smaller version of the same file block is shown with the bottom portion torn off and no closing bracket, labelled "interrupted write: 0 records recoverable". The right panel is headed "JSONL" and shows a file block of several separate record rows with no brackets, each row marked with a small tick and the whole annotated "each line independently valid". An arrow points from the first row to a small box labelled "process now" with a caption "records 2 to 2,000,000 still on disk". Beneath it, a second smaller block is shown with the bottom rows torn off, labelled "interrupted write: all complete lines recoverable".
Required elements: Opening and closing brackets on the left file, no brackets on the right, a whole-file bracket annotation on the left, per-line tick marks on the right, a torn interrupted version under each panel with its recovery label, the process-now arrow on the right.
Style: Clean educational illustration, professional, uncluttered, high contrast, flat vector, abstract record rows rather than legible data.
Layout: Two equal panels side by side with a thin vertical divider, each with a main block above and the interrupted variant below.
Text labels: "JSON array", "JSONL", "must read all of it before any record is usable", "each line independently valid", "process now", "records 2 to 2,000,000 still on disk", "interrupted write: 0 records recoverable", "interrupted write: all complete lines recoverable".
Aspect ratio: 16:9
Accessibility: Convey validity and recoverability through tick marks, torn edges, and text labels rather than colour alone.
Avoid: Real JSON text, code screenshots, tiny type, logos, watermarks, clutter.
Alt text: Comparison showing a JSON array that must be fully read before any record is usable and is unrecoverable if truncated, against JSONL where every line is independently valid, processable immediately, and recoverable up to the last complete line.
END IMAGE PROMPT]

When JSON is still correct. Use it for configuration files, for a single structured object, for API request and response bodies, and for anything a human will read and edit by hand. Use JSONL for collections of records, especially large ones, and for anything a pipeline produces or consumes.

JSON versus Pickle, and Pickle as a security topic

Pickle is Python's built-in serialization format. It can store almost any Python object, including custom classes, which JSON cannot.

import pickle

with path.open("wb") as f:
    pickle.dump(documents, f)

with path.open("rb") as f:
    documents = pickle.load(f)

That convenience comes with three serious problems.

Unpickling executes code. This is the important one. The pickle format includes instructions that are run during loading, which means loading a pickle file can do anything your program can do: delete files, open network connections, exfiltrate credentials. Loading an untrusted pickle is equivalent to running an untrusted script.

The Python documentation states this plainly, and it is worth internalising rather than filing away. A pickle file downloaded from the internet, received from a user, or fetched from a model hub is untrusted input. Several real supply-chain incidents have involved exactly this, because some machine learning model formats are pickle based.

Never load a pickle you did not create yourself, on the machine that created it. There is no safe mode and no sandbox flag. If you need to accept serialized data from elsewhere, use a format that describes data rather than instructions.

It is not portable. A pickle written by one Python version may not load in another, and it cannot be read by any other language. A pickled object referencing a class that has since been renamed or moved fails to load, which means your archived data is coupled to your code layout.

It is not inspectable. You cannot open a pickle in a text editor to see what went wrong. Debugging a bad record means writing a program.

The practical rule. Use JSON or JSONL for anything that crosses a boundary, meaning anything written to disk for later, sent between processes, or shared with anyone. Use pickle only for short-lived local caching where you wrote the file, you will read it in the same environment, and losing it costs nothing. If you are unsure, you want JSONL.

For records this is not even a compromise, because Module 2's records serialize cleanly:

from dataclasses import asdict

write_jsonl(output_path, (asdict(doc) for doc in documents))

Note the generator expression rather than a list. That is the subject of the next lesson.