Part 23 · Data Pipelines and Streaming Processing

Lesson 5: Tabular Tools

7 min read·9 Sept 2026

Pandas as plumbing only

Pandas is excellent at a specific set of jobs and heavily overused for everything else. In a pipeline, the useful framing is: use it to move and reshape tabular data, then get out.

The five operations worth using it for:

python

import pandas as pd

# Load
df = pd.read_csv("metadata.csv")
df = pd.read_json("records.jsonl", lines=True)
df = pd.read_parquet("records.parquet")

# Filter
recent = df[df["ingested_at"] > "2026-01-01"]

# Join
enriched = df.merge(sources, on="source_id", how="left")

# Deduplicate
unique = df.drop_duplicates(subset=["content_hash"], keep="first")

# Write
unique.to_parquet("clean.parquet", index=False)

Note lines=True for JSONL, and index=False on write, which stops pandas adding an unnamed index column that confuses every later reader.

Where the boundary is. Convert out of pandas before doing per-row business logic.

python

# Avoid: business logic inside pandas
df["cleaned"] = df["content"].apply(lambda t: apply_pipeline(t, CLEANING_PIPELINE))
df["is_valid"] = df.apply(lambda row: validate(row["cleaned"], row["source"]), axis=1)

# Prefer: pandas for the table work, records for the logic
df = pd.read_json(path, lines=True).drop_duplicates(subset=["content_hash"])

documents = (Document(**row) for row in df.to_dict("records"))
cleaned = clean_documents(documents)
write_jsonl(output, (asdict(doc) for doc in cleaned))

Three reasons the second form is better. df.apply with axis=1 is row-by-row Python with substantial overhead per row, so it is slower than a plain loop rather than faster. The logic inside a lambda cannot be unit tested, while clean_documents can. And the typed record gives you field checking, while row["cotnent"] gives you a KeyError at runtime, which is exactly the argument from Module 2.

Common pandas traps worth naming.

Chained assignment such as df[df.x > 1]["y"] = 0 may modify a copy rather than the original, and whether it does has changed between versions. Use .loc for assignment.

NaN is a float, so an integer column containing a missing value becomes a float column, and identifiers silently turn into 4.0. Read identifier columns with dtype=str.

Memory use is typically several times the file size, so a two gigabyte CSV can need eight gigabytes to load. This is the reason pandas is a poor fit for the corpus in this module and a fine fit for the metadata about it.

read_csv infers types by sampling, so a column that looks numeric in the first thousand rows and contains text at row fifty thousand produces a mixed column or an error late in the read.

[IMAGE PROMPT M4-4
Purpose: Show a decision path for choosing between streaming records, pandas, Polars, and DuckDB based on data size and operation type.
Visual type: Decision tree flowchart.
Prompt: A clean educational decision-tree flowchart reading top to bottom. The root box asks "Is the data tabular with columns and rows?". A "no" branch leads left to a terminal box labelled "stream records with generators" and a sub-label "documents, text, JSONL". A "yes" branch leads down to a second decision box asking "Does it fit comfortably in memory?". Its "yes" branch leads to a terminal box labelled "pandas" with the sub-label "load, filter, join, dedupe, write". Its "no" branch leads to a third decision box asking "Is the work mostly aggregation and joins?". That box's "yes" branch leads to a terminal box labelled "DuckDB" with the sub-label "SQL over files, out of core". Its "no" branch leads to a terminal box labelled "Polars" with the sub-label "lazy dataframe, larger than memory". A note beside the pandas terminal reads "convert to records before per-row logic".
Required elements: Three decision boxes with their exact questions, four terminal boxes with sub-labels, yes and no labels on every branch, the note beside the pandas terminal.
Style: Clean educational illustration, professional, uncluttered, high contrast, flat vector, diamond or rounded decision shapes distinct from rectangular terminals.
Layout: Vertical flow reading top to bottom with branches spreading left and right.
Text labels: "Is the data tabular with columns and rows?", "Does it fit comfortably in memory?", "Is the work mostly aggregation and joins?", "yes", "no", "stream records with generators", "documents, text, JSONL", "pandas", "load, filter, join, dedupe, write", "DuckDB", "SQL over files, out of core", "Polars", "lazy dataframe, larger than memory", "convert to records before per-row logic".
Aspect ratio: 4:3
Accessibility: Label every branch with yes or no in text, and use distinct shapes for decisions and terminals rather than colour alone.
Avoid: Vendor logos, decorative elements, tiny text, watermarks, clutter.
Alt text: Decision tree for choosing a data tool, routing non-tabular data to streaming generators, in-memory tabular data to pandas, larger-than-memory aggregation work to DuckDB, and other larger-than-memory dataframe work to Polars.
END IMAGE PROMPT]

When to reach for Polars or DuckDB

[VOLATILE: this section compares tools whose capabilities are converging quickly. Verify current claims about lazy evaluation, memory behaviour, and interoperability before publishing.]

Polars is a dataframe library written in Rust with a different design from pandas. Two differences matter for pipeline work.

It has a lazy API, so you describe a sequence of operations and it plans them before executing, which allows it to skip reading columns you never use and to fuse operations rather than materialising each step.

python

import polars as pl

result = (
    pl.scan_parquet("records/*.parquet")     # scan, not read: nothing loaded yet
    .filter(pl.col("status") == "valid")
    .group_by("source_domain")
    .agg(pl.len().alias("document_count"))
    .sort("document_count", descending=True)
    .collect()                                # now it runs
)

scan_parquet and collect are the lazy pair, and the shape should look familiar from Lesson 2. It is the same idea applied to tables: describe the pipeline, execute once, avoid materialising intermediates.

The second difference is that Polars can process data larger than memory in streaming mode, where pandas cannot.

DuckDB is an embedded analytical database that runs inside your process, with no server to install or manage. It queries files directly with SQL.

python

import duckdb

result = duckdb.sql("""
    SELECT source_domain, count(*) AS document_count
    FROM 'records/*.parquet'
    WHERE status = 'valid'
    GROUP BY source_domain
    ORDER BY document_count DESC
""").df()

The file path in the FROM clause is not a typo. DuckDB reads Parquet, CSV, and JSON files directly, without an import step, and it handles datasets larger than memory by design.

Choosing between the three.

Reach for pandas when the data fits in memory, the ecosystem matters, and someone else will read your code. It remains the most widely known, which is a genuine engineering consideration.

Reach for Polars when data is larger than memory, when performance matters, or when you want the lazy pipeline shape for dataframe work.

Reach for DuckDB when the work is naturally expressed as SQL, especially aggregations and joins across several files, or when you want to query intermediate Parquet output without loading it.

Reach for none of them when your data is documents rather than tables. This is the mistake most worth avoiding in this module. A corpus of two million text files is not a dataframe, and loading it into one converts a constant-memory streaming problem into an out-of-memory problem for no benefit. Use the record and generator pipeline from Lessons 2 and 3, and use a tabular tool for the metadata about the corpus rather than the corpus itself.

A note on Parquet. All three tools read and write it, and it is worth knowing why it keeps appearing. Parquet stores data by column rather than by row, compresses well, and preserves types, so reading two columns from a hundred-column file reads only those two. For intermediate pipeline output that will be queried repeatedly, it is usually a better choice than CSV or JSONL. For streaming append-only output written record by record, JSONL remains simpler, which is why this module uses JSONL for pipeline output and mentions Parquet for analysis.

Concept check. Your pipeline produces 40 million chunk records with an identifier, a document identifier, a text field, and an embedding. You need to count chunks per source document and find documents with more than 500 chunks. Which tool, and why?

Answer

DuckDB, or Polars in lazy mode. The work is a group-by and a filter over a column, which is exactly what analytical engines are built for, and 40 million rows containing embeddings will not fit comfortably in memory as a pandas dataframe.

The important part of the answer is what you do not load. The query needs only the document identifier column, so writing the chunks as Parquet lets the engine read that one column and ignore the text and embedding entirely. The same query against JSONL would parse every field of every record to reach one of them.

Streaming with generators would also work and would use almost no memory, but you would be writing the grouping logic by hand for something SQL expresses in a line. Aggregation across a large table is the case where a tabular tool genuinely earns its place.