Lesson 2: Lazy Processing

The problem: memory that scales with the corpus

Here is the pipeline from Module 3, written the way most people write it first.

def process_corpus(folder: Path, output: Path) -> None:
    raw_documents = [p.read_text(encoding="utf-8") for p in folder.iterdir()]
    cleaned = [clean_text(raw) for raw in raw_documents]
    chunks = [chunk for text in cleaned for chunk in chunk_text(text)]
    write_jsonl(output, chunks)

Four lines, readable, correct, and unusable on the real corpus.

Every line builds a complete list before the next line starts. After line one, every document is in memory. After line two, every document is in memory twice, once raw and once cleaned. After line three, every chunk of every document is in memory as well. Peak memory is several times the corpus size, and the corpus is forty gigabytes.

The fix is not to process less. It is to stop holding everything at once.

Iterators and generators

An iterator is an object that produces values one at a time. A generator is the easy way to create one, using yield instead of return.

def read_lines(path: Path) -> Iterator[str]:
    with path.open(encoding="utf-8") as f:
        for line in f:
            yield line.rstrip("\n")

The mechanics matter, so read carefully. Calling read_lines(path) runs none of the function body. It returns a generator object immediately. The body runs only when something asks for a value, and it runs up to the first yield, hands that value over, and then pauses with all its local state intact. The next request resumes exactly where it left off.

lines = read_lines(path)        # nothing has been read yet
first = next(lines)             # opens the file, reads one line, pauses
second = next(lines)            # resumes, reads one more line, pauses

Compare with the eager version:

def read_lines_eager(path: Path) -> list[str]:
    with path.open(encoding="utf-8") as f:
        return [line.rstrip("\n") for line in f]

This reads the entire file before returning anything. On a four gigabyte file it needs four gigabytes of memory and a noticeable wait before the first line is available.

One-pass semantics, which is the trap. A generator can be consumed once. After that it is exhausted and yields nothing.

lines = read_lines(path)
count = sum(1 for _ in lines)        # consumes everything
first = next(lines, None)            # None, the generator is exhausted

This bites when you pass a generator to two functions, or iterate it twice in a loop, and the second use silently sees nothing. A list would have worked, which is why the bug is confusing: the code is correct for a list and wrong for a generator, and the annotation is often what tells you which you have.

The rules that avoid it. If you need the data twice, materialise it once with list(...) and accept the memory cost deliberately. If you need to know the length, you must consume it, so count while processing rather than calling len(). And annotate return types as Iterator[X] rather than list[X] so callers know what they are receiving.

Generator expressions are the compact form, using parentheses instead of brackets.

lengths = (len(line) for line in read_lines(path))     # lazy
lengths = [len(line) for line in read_lines(path)]     # eager, builds the whole list

One character of difference, and an entirely different memory profile.

Generator pipelines at constant memory

Chain generators and each stage pulls one item from the stage before it. Nothing accumulates.

from collections.abc import Iterator


def read_documents(folder: Path) -> Iterator[RawDocument]:
    for path in folder.rglob("*.html"):
        result = read_document_text(path)
        if result is None:
            continue
        text, encoding = result
        yield RawDocument(path=path, text=text, encoding=encoding)


def clean_documents(docs: Iterator[RawDocument]) -> Iterator[Document]:
    for raw in docs:
        cleaned = apply_pipeline(raw.text, CLEANING_PIPELINE)
        yield build_document(raw.path, cleaned)


def chunk_documents(docs: Iterator[Document]) -> Iterator[Chunk]:
    for doc in docs:
        for index, text in enumerate(chunk_text(doc.content, size=500)):
            yield Chunk(document_id=doc.id, index=index, text=text)


def process_corpus(folder: Path, output: Path) -> None:
    documents = read_documents(folder)
    cleaned = clean_documents(documents)
    chunks = chunk_documents(cleaned)
    write_jsonl(output, (asdict(chunk) for chunk in chunks))

The first three lines of process_corpus do no work at all. They connect generators. The work happens inside write_jsonl, when it iterates and pulls the first chunk, which pulls a document, which pulls a file. One document is in memory at a time, and the pipeline runs in roughly the same memory whether the corpus holds two hundred files or two million.

Compare this to the eager version at the start of the lesson. The structure is nearly identical, which is the point: the shape from Module 3 survives, and laziness is a change of mechanism rather than of design.

[IMAGE PROMPT M4-2
Purpose: Contrast eager list-building against a lazy generator pipeline, showing how many items exist in memory at each stage.
Visual type: Two-panel process comparison with memory occupancy shown.
Prompt: A clean educational side-by-side comparison. The left panel is headed "Eager" and shows four stacked stage boxes labelled "read", "clean", "chunk", and "write", connected by downward arrows. Beside each of the first three stages sits a tall container labelled "all items in memory", each drawn full. A memory gauge on the far left of the panel is drawn nearly full and labelled "peak memory grows with corpus size". The right panel is headed "Lazy" and shows the same four stage boxes connected by downward arrows, but beside each stage sits a small container holding a single small square, labelled "one item". A memory gauge on the far right is drawn nearly empty and labelled "peak memory stays flat". A caption spanning both panels beneath reads "same pipeline shape, different mechanism".
Required elements: Four identically named stages in both panels, full containers on the left and single-item containers on the right, a memory gauge in each panel with its label, the shared caption beneath.
Style: Clean educational illustration, professional, uncluttered, high contrast, flat vector, gauges drawn as simple vertical bars with a fill level.
Layout: Two equal panels side by side, each reading top to bottom, shared caption centred beneath both.
Text labels: "Eager", "Lazy", "read", "clean", "chunk", "write", "all items in memory", "one item", "peak memory grows with corpus size", "peak memory stays flat", "same pipeline shape, different mechanism".
Aspect ratio: 16:9
Accessibility: Show memory difference through container fill level and explicit text labels rather than colour alone.
Avoid: Code screenshots, decorative icons, tiny text, logos, watermarks, clutter.
Alt text: Comparison of an eager pipeline holding every item in memory at each of four stages against a lazy generator pipeline holding one item per stage, with memory gauges showing growing versus flat memory use.
END IMAGE PROMPT]

What breaks laziness accidentally. Certain operations must consume everything, and inserting one into a lazy pipeline silently restores the memory problem.

sorted(chunks)              # must see every item before yielding the first
list(chunks)                # explicit materialisation
len(list(chunks))           # same
max(chunks, key=...)        # must see everything
random.shuffle(chunks)      # requires a list

None of these are wrong. They are sometimes necessary, and sorting genuinely requires all the data. The point is to know when you have done it, and to do it deliberately rather than by reflex. If your pipeline must sort forty gigabytes, that is a design decision requiring an external sort or a database, not a line of Python you write without noticing.

sum, min, max, any, and all consume the iterator but do not store it, so they are safe on memory even though they exhaust the generator.

itertools for batching, chunking, and windowing

itertools provides lazy building blocks. Four are worth knowing now.

islice takes a slice without materialising.

from itertools import islice

first_ten = list(islice(documents, 10))     # for sampling during development

This is the correct way to test a pipeline on a subset. documents[:10] does not work on a generator at all.

batched groups items into fixed-size tuples. Available in Python 3.12 and later.

from itertools import batched

for batch in batched(chunks, 100):
    embeddings = embed_many(batch)          # one API call per 100 chunks
    write_jsonl(output, embeddings)

Batching is the single most common need in an AI pipeline, because API calls have per-request overhead and per-request limits. On earlier Python versions, write it yourself:

def batched_fallback(iterable: Iterable[T], size: int) -> Iterator[tuple[T, ...]]:
    iterator = iter(iterable)
    while batch := tuple(islice(iterator, size)):
        yield batch

Note that the batch is materialised, which is correct and deliberate. A batch of 100 in memory is fine. Two million is not.

chain joins iterables end to end.

from itertools import chain

all_documents = chain(read_documents(folder_a), read_documents(folder_b))

Nothing is loaded. chain.from_iterable flattens one level of nesting lazily, which replaces the nested comprehension used in the eager example earlier.

Sliding windows for overlapping chunks, which matters because a chunk boundary can split a sentence.

from itertools import islice
from collections import deque


def sliding_window(iterable: Iterable[T], size: int) -> Iterator[tuple[T, ...]]:
    iterator = iter(iterable)
    window = deque(islice(iterator, size), maxlen=size)
    if len(window) == size:
        yield tuple(window)
    for item in iterator:
        window.append(item)                  # oldest falls off automatically
        yield tuple(window)

The deque with maxlen from Module 2 does the work here, holding only size items regardless of input length.

One warning about groupby. itertools.groupby groups only consecutive equal items, so it requires sorted input to behave the way people expect. On unsorted data it produces many small groups rather than the grouping you intended, and it produces them without complaint.

Stream by default, materialize only when required

The principle for the rest of the course: process data as a stream unless you have a specific reason to hold it.

Materialise when the data is genuinely small and bounded, such as a configuration file or a batch. Materialise when you need multiple passes and re-reading is more expensive than storing. Materialise when an operation genuinely requires all the data, such as sorting or shuffling. Materialise when the data must be indexed by position.

Stream everywhere else, which is most places.

Practical guidance for writing streaming code.

Return Iterator[X] from functions producing many items, and annotate it so callers know.

Accept Iterable[X] in functions consuming them, rather than list[X], so a caller can pass a generator, a list, or anything else iterable. Iterable is the wider type and the correct one for a parameter.

Keep the consuming loop in one place, usually the outermost function, so there is one obvious point where the work actually happens.

Do not open a file in one function and iterate it in another without a context manager spanning both, or you will leak file handles. The with statement inside the generator, as written in read_lines, handles this correctly because the generator's scope holds the file open until it is exhausted or closed.

Predict the output.

def numbers() -> Iterator[int]:
    print("starting")
    for n in range(3):
        print(f"yielding {n}")
        yield n


gen = numbers()
print("created")
first = next(gen)
print(f"got {first}")

Answer

created
starting
yielding 0
got 0

Nothing inside the function runs when numbers() is called, which is why created prints first. The body starts only at next(gen), runs up to the first yield, and pauses there. The remaining two iterations never happen because nothing asked for them.

This ordering is the whole idea behind laziness, and seeing it printed is usually the moment it becomes concrete.