Part 22 · Data Pipelines and Streaming Processing

Lesson 4: Multimodal Inputs

5 min read·9 Sept 2026

Reading images

Text is not the only input. Recipe pages contain photographs, scanned cookbooks arrive as images, and models increasingly accept images directly.

The library for image handling is Pillow, imported as PIL.

python

from PIL import Image

with Image.open(path) as img:
    width, height = img.size
    fmt = img.format              # 'JPEG', 'PNG', 'WEBP'
    mode = img.mode               # 'RGB', 'RGBA', 'L' for greyscale

Image.open is lazy about pixel data, reading only the header until pixels are actually needed, so checking dimensions across a large folder is cheap.

Validate before processing. Images from a scrape include corrupt files, wrong extensions, and files large enough to exhaust memory when decoded.

python

MAX_PIXELS = 50_000_000


def load_image(path: Path) -> Image.Image | None:
    """Open and validate an image, returning None if it is unusable."""
    try:
        img = Image.open(path)
        img.verify()                      # checks integrity, then invalidates the object
    except Exception:
        return None

    img = Image.open(path)                # verify() consumed it, reopen to use
    width, height = img.size
    if width * height > MAX_PIXELS:
        return None

    return img.convert("RGB")             # normalise mode for consistent downstream code

Two details. verify() invalidates the image object, so you must reopen it, which is a genuine oddity of the library rather than a mistake here. And the pixel limit matters because a decoded image needs roughly width times height times channels bytes, so a 20,000 by 20,000 pixel image needs over a gigabyte regardless of its file size. This class of file is sometimes malicious, known as a decompression bomb, and Pillow has its own warning threshold you can configure.

Resizing before sending anywhere.

python

def prepare_for_model(img: Image.Image, max_dimension: int = 1568) -> Image.Image:
    """Downscale so the longest side fits, preserving aspect ratio."""
    img.thumbnail((max_dimension, max_dimension), Image.Resampling.LANCZOS)
    return img

thumbnail modifies in place, preserves aspect ratio, and never enlarges. Resizing before sending matters for cost as well as limits, because image inputs are usually billed by resolution. [VOLATILE: the 1568 pixel default reflects a common provider guideline. Check the current recommended maximum for your provider before publishing.]

Base64 encoding and size limits

Images are bytes and most APIs accept JSON, so images are sent as base64, a text encoding of binary data.

python

import base64
from io import BytesIO


def image_to_base64(img: Image.Image, fmt: str = "JPEG", quality: int = 85) -> str:
    buffer = BytesIO()
    img.save(buffer, format=fmt, quality=quality)
    return base64.b64encode(buffer.getvalue()).decode("ascii")

BytesIO is an in-memory file, which lets you save to bytes without touching disk.

The size arithmetic you must know. Base64 uses four characters for every three bytes, so encoding inflates size by approximately 33 percent. A two megabyte JPEG becomes roughly 2.7 megabytes of text.

This matters in three ways. Provider limits usually apply to the encoded payload rather than the original file, so a file just under the limit can be rejected after encoding. Base64 strings are held in memory as Python strings, and a batch of fifty images is a substantial amount of memory. And the encoded string travels in your request body, so it affects latency.

Never put base64 image data in a log line. A single log statement containing an encoded image writes megabytes of noise and can fill a disk. The log_call decorator from Module 3 logged types rather than values for precisely this reason.

A practical helper with limits enforced:

python

MAX_ENCODED_BYTES = 5 * 1024 * 1024


def encode_for_model(path: Path) -> str | None:
    img = load_image(path)
    if img is None:
        return None

    img = prepare_for_model(img)

    for quality in (85, 70, 55):
        encoded = image_to_base64(img, quality=quality)
        if len(encoded) <= MAX_ENCODED_BYTES:
            return encoded

    return None

Reducing quality stepwise until the payload fits is more useful than failing, and returning None when it cannot fit lets the caller record a rejection with a reason rather than crashing.

Reading PDFs and extracting pages

PDFs are the hardest common input, because a PDF describes visual appearance rather than semantic structure. There is no paragraph object and no table object, only text positioned at coordinates. Every extraction library reconstructs structure by guessing from layout.

The library landscape. [VOLATILE: this section describes the state of the ecosystem at time of writing. Verify library names, versions, and licence terms before publishing.]

pypdf is pure Python, widely installed, permissively licensed, and adequate for straightforward text extraction. PyMuPDF, imported as fitz, is substantially faster and better at layout, but is AGPL licensed, which requires either open sourcing your own code or buying a commercial licence. That licence point is a real engineering constraint that catches teams late, so check it before building on it. pdfplumber is the usual choice when tables matter. For scanned documents, no text layer exists at all and you need OCR or a vision model.

Basic extraction with pypdf:

python

from pypdf import PdfReader


def extract_pdf_text(path: Path) -> Iterator[tuple[int, str]]:
    """Yield (page_number, text) for each page, one page at a time."""
    reader = PdfReader(path)
    for page_number, page in enumerate(reader.pages, start=1):
        text = page.extract_text() or ""
        if text.strip():
            yield page_number, text

Note the generator, so a 900 page document does not arrive as one enormous string. Note also or "", because extraction returns None for pages with no text layer, and yielding page numbers alongside text so a chunk can cite its source page.

Detecting a scanned PDF. A document whose pages produce almost no text is an image-based PDF, and processing it as text produces empty records:

python

def has_text_layer(path: Path, sample_pages: int = 3) -> bool:
    reader = PdfReader(path)
    pages = reader.pages[:sample_pages]
    total = sum(len((page.extract_text() or "").strip()) for page in pages)
    return total > 100

Sampling the first few pages is enough, and it costs far less than extracting the whole document to discover it is empty. Route documents failing this check to an OCR path or reject them with a specific reason rather than letting them through as empty text.

Page extraction for large documents. Rendering pages to images lets you send them to a vision model, which is often the better answer for complex layouts:

python

def render_page_to_image(path: Path, page_number: int, dpi: int = 150) -> bytes:
    """Render one PDF page as PNG bytes. Requires PyMuPDF."""
    import fitz

    with fitz.open(path) as doc:
        page = doc[page_number - 1]
        pixmap = page.get_pixmap(dpi=dpi)
        return pixmap.tobytes("png")

DPI is the tradeoff between legibility and size. 150 is usually enough for text, and 300 produces much larger images for modest gain unless the source is small print.

The practical decision. Text layer present and layout simple means extract text. Text layer present and tables matter means pdfplumber. No text layer, or layout that extraction mangles, means render pages and use a vision model. Deciding per document rather than per corpus is worth the small cost of the check.