Summary

Paths are objects, not strings. pathlib handles joining, suffixes, and traversal correctly across platforms, and its directory walkers are lazy, which matters before you have read a single file.

Encodings are the first thing that breaks on someone else's data. State the encoding explicitly, decide deliberately between strict, replace, and detect, record which was used, and normalise Unicode before hashing so that visually identical text deduplicates.

JSONL is the default interchange format for pipelines because each line is independently valid. That single property gives you streaming reads, cheap appends, usable partial files after a crash, per-record error isolation, and compatibility with ordinary text tools. Pickle is a security topic rather than a format choice, because loading one executes code.

Generators produce values one at a time and pause between them, so a chained pipeline of generators processes a corpus in constant memory regardless of its size. The pipeline shape from Module 3 survives the change. Know which operations break laziness, and use itertools for batching, chaining, and windowing without materialising.

A long job must survive interruption. Checkpoint after writing rather than before, so a crash duplicates work instead of losing data, then make writes idempotent so the duplication is harmless. Write files atomically through a temporary file and a rename. Produce a run report every time, including when the run fails, and a rejection report that explains every document that did not make it.

Images need validating and resizing before encoding, and base64 inflates payloads by roughly a third. PDFs describe appearance rather than structure, so check for a text layer before extracting and route scanned documents elsewhere.

Tabular tools are for tables. Use pandas as plumbing and convert to records before applying business logic. Reach for Polars or DuckDB when data outgrows memory or the work is naturally aggregation. Do not put a document corpus in a dataframe.

Key takeaways

• Always state the encoding, and record which one was actually used
• Normalise Unicode before hashing, or deduplication silently misses matches
• JSONL is streamable, appendable, and survives a crash, which is why pipelines default to it
• Never load a pickle you did not write yourself
• Calling a generator function runs none of it, which is the whole point
• A generator can be consumed once, and the second use silently sees nothing
• Sorting, shuffling, and list() end laziness, so do them deliberately
• Write the output first and the checkpoint second, then make the write idempotent
• Flush after writing, or a hard kill discards what was buffered
• A run report is how you tell a normal run from a broken one
• Base64 adds about a third to the payload, and never belongs in a log line
• A PDF without a text layer needs a different path, so detect it early
• Do not put a document corpus in a dataframe

Common mistakes to remember

• Building paths with string concatenation
• Reading text without an explicit encoding
• Using errors="ignore" and losing content silently
• Loading a whole JSON array when JSONL would stream
• Reaching for pickle because it was convenient
• Calling len() on a generator, or iterating it twice
• Annotating a generator-returning function as list[X]
• Accepting list[X] as a parameter when Iterable[X] would do
• Checkpointing before writing, so a crash loses records
• Forgetting to flush, so a kill discards buffered output
• Truncating the output file on a resumed run instead of appending
• Decoding an enormous image without a pixel limit
• Logging a base64 image payload
• Extracting text from a scanned PDF and producing empty records
• Using df.apply(axis=1) for business logic that belongs in a tested function
• Losing identifier columns to float conversion because of a missing value