Lesson 1: From a messy ticket to tokens a model can count

Module 5 · Lesson 1 · About 40 minutes · Needs: Module 1 (Python strings and regular expressions)

The situation

Priya wants to know what customers complain about most, so Dev exports a week of tickets and counts the words. The count is useless: refund, Refund and refund. count as three words. Tickets from the web form carry <p> tags and & where the customer typed &. One has an apostrophe that looks normal but matches nothing.

Nothing is wrong with the counting. The text was never made ready. This lesson takes one messy ticket from raw string to tokens, the pieces search and language models work with, and shows what each step keeps and loses.