Lesson 2: Functions as Values

Higher-order functions

In Python a function is a value. You can assign it to a name, put it in a list, pass it to another function, and return it from one. A function that takes or returns another function is a higher-order function.

Note the difference between referring to a function and calling it.

clean_text            # the function itself, a value
clean_text(raw)       # the result of calling it

Leaving off the parentheses is how you pass a function somewhere. The CLEANING_PIPELINE list in Lesson 1 was a list of function values, and apply_pipeline was a higher-order function that took them.

sorted(key=...) is the one you will use most. The key parameter takes a function that is applied to each item to decide what to sort by.

# Sort documents by length, shortest first
by_length = sorted(documents, key=len)

# Sort by a field
by_date = sorted(documents, key=lambda doc: doc.ingested_at)

# Sort by a computed value, longest first
by_word_count = sorted(documents, key=count_words, reverse=True)

The same key idea appears in min, max, and itertools.groupby, so it is worth being comfortable with.

For sorting by an attribute or a dictionary key, the operator module is faster and clearer than a lambda:

from operator import attrgetter, itemgetter

by_date = sorted(documents, key=attrgetter("ingested_at"))
by_score = sorted(results, key=itemgetter("score"), reverse=True)

# Sort by two things at once: status, then date within status
by_status_then_date = sorted(documents, key=attrgetter("status", "ingested_at"))

That last form, sorting by several keys at once, is worth remembering because writing it by hand is fiddly.

map and filter. These apply a function across an iterable, or keep items for which a function returns true.

lengths = map(len, documents)
valid = filter(lambda doc: doc.status == "valid", documents)

Both return lazy iterators rather than lists, so nothing is computed until you consume them. In most cases a comprehension is clearer:

lengths = [len(doc) for doc in documents]
valid = [doc for doc in documents if doc.status == "valid"]

Use map when you already have a named function and the comprehension would just wrap it: map(clean_text, raw_documents) reads better than [clean_text(r) for r in raw_documents]. Use a comprehension whenever there is any transformation expression involved, and avoid filter with a lambda entirely, since the comprehension form is shorter and more readable.

lambda, and when not to use it

A lambda is an anonymous function written inline. It is limited to a single expression and cannot contain statements.

sorted(documents, key=lambda doc: doc.word_count)

That is the good use: a tiny throwaway function passed directly as an argument, where naming it would add a line and no clarity.

When not to use one.

Do not assign a lambda to a name. If it deserves a name it deserves to be a function, and the def form gives it a proper name in tracebacks and allows a docstring.

score = lambda doc: doc.word_count * doc.relevance      # avoid

def score(doc: Document) -> float:                       # prefer
    return doc.word_count * doc.relevance

Do not write a long lambda. Anything with a conditional expression and arithmetic is past the point where an inline function helps.

key=lambda d: (d.priority, -d.score if d.score else 0, d.title.lower())   # too much

Extract it, name it, and the sort call becomes readable again.

Do not use a lambda where a named function already exists or where the operator module has one, as shown above.

And remember lambdas cannot be typed usefully. lambda doc: doc.word_count gives mypy nothing to check, whereas a def with annotations does.

Closures for configuration and factories

A closure is a function that remembers values from the scope where it was created. This sounds abstract and solves a very concrete problem.

Consider a validation function that needs a limit:

def is_long_enough(text: str, minimum: int) -> bool:
    return len(text.strip()) >= minimum

You cannot pass this to filter or put it in a pipeline, because those expect a function taking one argument. You could pass the minimum every time, but then it must be threaded through every layer that touches the function.

A closure solves it by building a configured function:

from collections.abc import Callable


def make_length_validator(minimum: int) -> Callable[[str], bool]:
    """Return a validator that checks text against this minimum length."""
    def validator(text: str) -> bool:
        return len(text.strip()) >= minimum
    return validator


is_long_enough = make_length_validator(minimum=50)
is_very_long = make_length_validator(minimum=5000)

is_long_enough("short")          # False
is_very_long(huge_document)      # True

make_length_validator is a factory, meaning a function that builds other functions. The inner validator closes over minimum, keeping access to it after the outer function has returned. Each call to the factory produces an independent function with its own captured value.

Now the validator fits anywhere a one-argument function is expected:

long_documents = [doc for doc in documents if is_long_enough(doc.content)]

[IMAGE PROMPT M3-3
Purpose: Show how a factory function produces separate configured functions, each holding its own captured value.
Visual type: Mechanism diagram showing one factory producing two distinct outputs.
Prompt: A clean educational diagram. On the left, a box labelled "make_length_validator(minimum)" with two inbound arrows on its left labelled "minimum = 50" and "minimum = 5000". Two outbound arrows lead to the right into two separate rounded boxes. The upper box is labelled "is_long_enough" and contains a smaller inner box labelled "captured: minimum = 50" plus a line reading "takes: text, returns: bool". The lower box is labelled "is_very_long" and contains a smaller inner box labelled "captured: minimum = 5000" plus the same "takes: text, returns: bool" line. A caption beneath the two output boxes reads "same code, separate captured values, independent functions".
Required elements: One factory box with two differing inputs, two distinct output function boxes, a visible captured-value compartment inside each output, matching signature lines, the caption beneath.
Style: Clean educational illustration, professional, uncluttered, high contrast, flat vector, clear nesting of the captured-value boxes.
Layout: Left to right, factory on the left, two stacked output functions on the right, caption centred beneath the outputs.
Text labels: "make_length_validator(minimum)", "minimum = 50", "minimum = 5000", "is_long_enough", "is_very_long", "captured: minimum = 50", "captured: minimum = 5000", "takes: text, returns: bool", "same code, separate captured values, independent functions".
Aspect ratio: 16:9
Accessibility: Show the captured value as a labelled inner compartment so the difference is textual, not colour based.
Avoid: Code screenshots, decorative icons, tiny text, logos, watermarks.
Alt text: Diagram showing one factory function called with two different minimum values, producing two independent validator functions each holding its own captured minimum.
END IMAGE PROMPT]

A practical use: configuring a pipeline. Lesson 1's cleaning pipeline was a fixed list. With factories, it becomes configurable without changing any transform:

def make_truncator(max_chars: int) -> TextTransform:
    def truncate(text: str) -> str:
        return text[:max_chars]
    return truncate


def build_pipeline(*, max_chars: int, lowercase: bool) -> list[TextTransform]:
    transforms: list[TextTransform] = [strip_html, normalize_whitespace]
    if lowercase:
        transforms.append(str.lower)
    transforms.append(make_truncator(max_chars))
    return transforms


pipeline = build_pipeline(max_chars=10_000, lowercase=True)

Note str.lower used directly as a transform. Methods are functions too, and str.lower is a function taking a string and returning a string, which is exactly what the pipeline expects.

The classic closure trap. A closure captures the variable, not its value at the moment of creation.

validators = []
for limit in [10, 100, 1000]:
    validators.append(lambda text: len(text) >= limit)

[v("x" * 50) for v in validators]
# [False, False, False]

All three closures reference the same limit variable, which by the time they run holds 1000. The fix is to capture the value explicitly with a default argument, which is evaluated at definition time:

for limit in [10, 100, 1000]:
    validators.append(lambda text, limit=limit: len(text) >= limit)

Using a factory function avoids the problem entirely, since each call creates a fresh scope. That is one more reason to prefer factories over inline lambdas in loops.