Module 1: The problem with multimodal search

About 45 minutes · Needs: nothing (Module 0 gives the tour) · Code: samples/segments.json, samples/questions.jsonl, samples/build_test_video.py, originals/v1/video-rag-old/README.md

Where this fits

This module is about the problem, not yet about a solution. The highlighted parts are the test video and its 26 questions, the two signals the first design pulls out of a video (frames and speech), and the author's original code, which promised to answer questions about any video. Everything later in the course is measured against what this module sets up: a video where some facts are only shown, some are only said, and some need both at once.

What it does

Picture a recorded architecture review. At 0:17 a slide shows a service map with one box outlined in red: Label Printer Service. Over that slide the presenter says "Now look at the service in the red box. That's the one we removed in March". Someone later asks "Which service did Tidewater remove in March?". The answer needs the picture (the name) and the speech (the month) at the same moment. Neither one alone is enough.

That is the whole problem of multimodal search in one example. A video carries two signals that run on one clock: what is shown and what is said. A question can need one of them, the other, or both together. A search system that treats them separately can find the right moment and still give a wrong answer, and the wrong answer can look convincing. This course starts there: "Most AI projects stop when the answer looks correct. This project starts there."

At a glance

Use cases and examples

Treat video as two timed signals

• This project: segment 5 (71.8 to 92.7 s) shows a clock and three tiers with no numbers; only the narration says "forty-five days". A system that only reads slides cannot answer s02.
• A hospital's recorded ward round: the chart on screen shows the dose, the doctor says which patient it is for. Losing either half makes the record dangerous.
• An e-learning platform: a lecturer writes a formula on a whiteboard and explains each symbol aloud. Students search for "what does lambda stand for".

Search by meaning, not keywords

• This project: c06 asks which "component" batches customer messages; the narration calls it "the step in the green box". The words differ, the meaning is the same.
• An online shop: "something warm for a winter wedding" should find a wool shawl that never uses the word "warm".

Label evidence by modality

• This project: the room name Harbor-4 is visual only, the presenter's name Priya Nair is speech only, the removed service is cross-modal.
• An insurer's claim videos: damage is shown, the time of the accident is said. Tag each claim question so you know which pipeline to test.

Define a good answer up front

• This project: c01's gold window is 17.333 to 36.133 s and the answer must contain "Label Printer". A reply without the name fails, however polite.
• A bank's call-centre search: an answer is only useful with the call's timestamp, so a supervisor can check it.

Test before you trust

• This project: Part 1 found the right moment for c01 at rank 1 and still answered "not mentioned".
• A SaaS help centre: a support bot that cites the right article but quotes last year's price looks correct in a demo and fails customers.

When not to bother. If your videos are talking heads with nothing useful on screen (a podcast with a static logo), a transcript search is enough and the picture adds nothing. If your videos are silent (security cameras, product demos without voice), frames alone carry everything. The problem in this module only bites when the two signals carry different facts about the same moment.

How it works

We walk through the test video and its questions: what is in them, why each question kind exists, and what a correct answer must contain. Every value comes from samples/segments.json, samples/questions.jsonl and the comparison report evals/video/reports/20261027T192400Z.json (one run on 27 Oct 2026, UTC stamp 20261027T192400Z, with the free-tier key).

Step 1: one clock, two tracks

The test video, test_video.mp4, is a 3.4-minute fictional architecture review: "Tidewater Freight: Shipment-Tracking Architecture Review". It lasts 202.467 s and has 12 segments. Each segment is one still slide with narration over it. The slide is the picture track, the narration is the sound track, and both share the same timeline. A segment starts with 0.5 s of silence, then the narration, then at least 1.5 s of silence, so every slide change lands in a pause. That matters later: a search result that covers 15 s of video should not cut a sentence in half by accident.

Why care? Every answer in this course is judged by time. A question has a gold window, the time range where its answer lives.

Step 2: what is shown and what is said

segments.json stores, for each segment, every string drawn on screen (screen_text) and the exact narration. Segment 2, arch_removed, shows a service map whose red box holds "Label Printer" "Service". Its narration says "the service in the red box. That's the one we removed in March". Neither field contains the whole fact. The screen has the name but not the month; the speech has the month but not the name.

The same pattern repeats on purpose. Segment 3 highlights the East bar at 640 ms while the speaker says "the bar I just highlighted ... where the geofence rollout will start". Segment 7 highlights Deepa Rao in yellow while the speaker says "the name highlighted in yellow is joining the rotation for the first time".

Step 3: why keywords are not enough

The simplest search is keyword search: find the passages that share words with the question. It fails here in three ways.

First, the words may not be there. "Which service did Tidewater remove in March?" shares "March" with the narration, but the answer, "Label Printer", is not spoken. A keyword search over the transcript finds the moment and cannot read the answer from it.

Second, the words may be somewhere else. The video contains keyword traps: "Geofence" is spoken in segment 3 and written in segments 6 and 12; "Tracking Service" is drawn on four slides. A question about the geofence rollout region (c03) matches three segments by keyword, and only one of them is right.

The same happens with ordinary words. v03 asks for the "total monthly cloud bill". The slide says "Monthly cloud cost by service" and "$18,400 per month". The only time "bill" is spoken is in "get the whole bill under fifteen thousand dollars by January". Part 1 answered v03 with "Based on the video, Finance has asked Tidewater to get their total monthly cloud bill under $15,000 by January [1]." The words matched; the fact did not.

Third, people use other words. c06 asks "Which component batches customer messages every thirty seconds?". The narration says "The step in the green box is where we batch messages". "Component" and "step" never meet in a keyword index.

Step 4: search by meaning

The alternative is semantic search, search by meaning. A model called an embedding model turns a piece of text (or an image, or a clip of audio) into a list of numbers, a vector, so that things with similar meaning get vectors that point in similar directions. "Which component batches customer messages" and "the step in the green box is where we batch messages" end up close together even though "component" and "step" are different words.

Meaning search fixes paraphrase. It does not fix the first two problems on its own: Part 1 already ranks by meaning, and the v03 answer above came out of it. If the name is only on screen and the month is only in speech, a meaning search over the transcript still finds a transcript chunk without the name, and a meaning search over the frames still has to guess which service map is meant. Which signals you embed, and whether you embed them together, decides what can be found. That choice is what Parts 1 and 2 disagree about.

Step 5: three kinds of evidence, and one kind of noise

The 26 questions are split by where the answer lives. Visual questions (7) have answers drawn on screen and never spoken: the room Harbor-4, the week 45 on-call engineer Aiko Tanaka, the $18,400 bill, the "5x" load test. Speech questions (7) have answers spoken and never shown: the presenter Priya Nair, 45-day retention, the December ninth review. Cross-modal questions (8) need both: the narration points at something ("the service in the red box", "the largest slice") and the name or value it points at is only on screen. Distractor questions (4) ask about a pizza workshop placed in the middle of the review, to check that a search can leave the main topic when asked to.

The kinds are a diagnostic tool. Failing every visual question means a broken picture path. Passing visual and speech questions but failing cross-modal ones means a binding problem: each signal is read, and they are never connected.

Step 6: near-duplicates make guessing impossible

Two pairs of slides are nearly identical. The two service maps (segments 2 and 6) differ only in the red-boxed service: Label Printer Service, then Geofence Worker. The two latency charts (segments 3 and 9) differ only in the East value and the week label: 640 ms in Week 37, 230 ms in Week 42. c01 and c02 can only be scored correct when the right service map is picked; c03 and c04 when the right chart is picked. A system that matches "a service map" is right half the time by luck, and the test is built so that luck does not pass.

Step 7: what a good answer needs

A good answer to a question about a video has three parts: a name or value (Label Printer Service), a time (where in the video it is), and a source (the retrieved moment the answer came from, so a person can check it). The checks follow from that. An answer passes when it contains any of the question's answer_contains strings, case-insensitive. A source is a hit when it overlaps a gold window.

Here are the two observed answers to c01 from the comparison run. Part 1 (the stitched design) answered: "Based on the provided video evidence, the specific name or type of the service removed in March is not mentioned; the speaker only refers to it as the service located "in the red box" on the service map [3]. It was removed because carrier portals were already performing the same job, which helped reduce deploy pipelines and late-night pages [3]." Part 2 (the redesign) answered: "Tidewater removed the Label Printer Service in March because the carrier portals were already performing the same job [0:24-0:33]." The first has a reason and a reference and no name. The second has all three parts.

Step 8: the answer that looks right

Look again at Part 1's c01 answer. It is fluent, it cites evidence, it gives the reason for the removal, and its top source overlaps the gold window: retrieval ranked the right moment first. A reader skimming it would call it good. It fails the one check that matters: it does not name the service.

The same pattern holds across all eight cross-modal questions in that run. Part 1 had a source in the gold window in its top 5 for 88% of them (7 of 8) and a passing answer for 0 of 8. That gap between "found the right place" and "gave the right answer" is what the course calls almost correct. Module 6 diagnoses it; Part 2 removes its cause. Part 2 also used ingest-time notes written by a Gemini model, so its answer rate mixes two changes; Module 10 explains why retrieval metrics are the fair comparison.

Code tour

This module's "code" is mostly data: the timeline, the questions, and the script that builds both. We read them in that order, then the original project's own description of its plan.

New concept: retrieval-augmented generation (RAG) A language model answers from what it learned in training, which knows nothing about your video. Retrieval-augmented generation first searches your own data for the passages most related to the question, then gives those passages to the model with the question and asks it to answer only from them. Retrieval decides what the model can see; the model only writes the sentence.

{
  "index": 2,
  "key": "arch_removed",
  "title": "Service map (red box: removed service)",
  "start_s": 17.333,
  "end_s": 36.133,
  "speech_start_s": 17.833,
  "speech_end_s": 34.613,
  "screen_text": ["Shipment tracking: service map", "Mobile App", "API Gateway", "Tracking", "Service",
                  "Route Planner", "Carrier Portals", "Label Printer", "Service", "Event Bus", ...],
  "narration": "Here's the current service map. ... Now look at the service in the red box. That's the one we
                removed in March, because the carrier portals were already doing the same job for us. ...",
  "facts": {
    "cross_modal": ["service removed in March = Label Printer Service (name on screen, 'removed in March' spoken)"],
    "visual_only": ["Tracking Service writes to Shipments DB"]
  }
}

This is segment 2 of samples/segments.json, shortened and reflowed (the ... marks cuts). start_s and end_s are frame-exact; speech_start_s is 0.5 s later and speech_end_s ends 1.52 s before the slide changes. The facts field is a note for humans: which fact is visual only, which is cross-modal.

# samples/build_test_video.py
SEGMENTS: list[Seg] = [
    # ...
    Seg(
        "arch_removed",
        "Service map (red box: removed service)",
        "Here's the current service map. Requests come in through the gateway and fan out from there. Now look at the "
        "service in the red box. That's the one we removed in March, because the carrier portals were already doing "
        "the same job for us. Retiring it cut one deploy pipeline and a lot of late-night pages.",
        lambda s: slide_arch(s, "Label Printer\nService", 2, 12),
        {
            "cross_modal": [
                "service removed in March = Label Printer Service (name on screen, 'removed in March' spoken)"
            ],
            "visual_only": ["Tracking Service writes to Shipments DB"],
        },
    ),

Code walkthrough

• SEGMENTS is the script's single source of truth: one Seg per slide, in order. The JSON file above is generated from it.
• The first two strings are the segment's key and title. The third is the narration, the exact text sent to the voice engine.
• lambda s: slide_arch(s, "Label Printer\nService", 2, 12) draws the slide with Pillow. The same slide_arch function draws segment 6 with "Geofence\nWorker" in the red box, which is how the near-duplicate pair is made: one function, one changed label.
• The dictionary at the end is the human note that becomes facts in segments.json.
• Nothing here mentions times: they are computed from the measured length of each narration clip.

New concept: modality A modality is a kind of signal: images, speech, written text, sound. A system is multimodal when it takes in more than one. In this course "visual" means what is on screen, "speech" means what is said, and "cross-modal" means a fact that only exists when you combine the two.

{"id": "c01", "question": "Which service did Tidewater remove in March?", "kind": "cross_modal", "gold": [{"start_s": 17.333, "end_s": 36.133, "segment": "arch_removed"}], "answer_contains": ["Label Printer"], "notes": "near-duplicate of arch_added: a frame-only match on the service map cannot tell the two apart"}
{"id": "v01", "question": "Which meeting room is the architecture review held in?", "kind": "visual", "gold": [{"start_s": 0.0, "end_s": 17.333, "segment": "title"}], "answer_contains": ["Harbor-4", "Harbor 4"], "notes": "answer is only on screen (never spoken)"}
{"id": "s02", "question": "How long are raw GPS pings kept?", "kind": "speech", "gold": [{"start_s": 71.8, "end_s": 92.733, "segment": "retention"}], "answer_contains": ["45 days", "forty-five days", "forty five days"], "notes": "answer is only spoken (never on screen)"}

Three lines of samples/questions.jsonl, one per kind that the course keeps coming back to. Each line is one JSON object: the question, its kind, one or more gold windows copied from segments.json, and the accepted answer strings. s02 accepts three spellings of the same number because a model may write digits or words.

# samples/build_test_video.py
    # cross-modal: the name/value is only on screen, what it means is only in the narration
    dict(
        id="c01",
        kind="cross_modal",
        seg=["arch_removed"],
        question="Which service did Tidewater remove in March?",
        answer_contains=["Label Printer"],
        screen_terms=["Label Printer"],
        speech_terms=["March"],
        notes="near-duplicate of arch_added: a frame-only match on the service map cannot tell the two apart",
    ),

Code walkthrough

• This is where c01 is defined. seg=["arch_removed"] names the gold segment by key; the build copies its exact start and end into the gold field, so the times cannot drift from the video.
• screen_terms and speech_terms say which words must be drawn and which must be spoken. For c01 the name is a screen term and "March" is a speech term: that is what makes it cross-modal.
• These two fields do not reach questions.jsonl. They feed a self-check in the build that fails if a screen-only term is ever spoken or a speech-only term is ever drawn. Module 6, on designing a test video that can fail, reads that check line by line.
• answer_contains is deliberately short: "Label Printer" passes "Label Printer Service" and "the label printer", and fails anything without the name.

New concept: embedding (meaning search) An embedding is a fixed-length list of numbers that a model produces for a piece of content. Models are trained so that related content gets nearby vectors. To search, you embed every stored item once, embed the question at query time, and return the items whose vectors are closest. Closeness, not shared words, decides the ranking.

<!-- originals/v1/video-rag-old/README.md -->
1.  **Ingestion**: 
    -   Frames are extracted every 5 seconds and embedded with **CLIP**.
    -   Audio is extracted and transcribed with **Whisper**.
    -   Transcript is chunked (512 tokens) and embedded with **MiniLM**.
    -   All vectors are stored in two separate Qdrant collections.
2.  **Retrieval**:
    -   Queries are encoded with both CLIP (visual) and MiniLM (textual).
    -   Top results from both collections are fused using **Reciprocal Rank Fusion**.
3.  **Synthesis**:
    -   Fused results are passed as context to Gemini 2.5 Flash Lite to generate the final answer.

Code walkthrough

• The author's original plan for Part 1, from originals/v1/video-rag-old (kept read-only). Line by line it is reasonable: a vision model for frames, a speech model for audio, a text model for the transcript, a vector database, a well-known way to merge rankings, a language model to write the answer.
• What it never says is where the picture and the speech meet. They are embedded by different models into "two separate Qdrant collections", and only their rankings are merged. That gap is invisible in the plan and decisive in the results. Module 2 walks the whole pipeline.
• The model named in step 3, Gemini 2.5 Flash Lite, now returns 404 for new users; the hardened code makes the model a setting.

# part1-video-rag-stitched/src/api/schemas.py
class Source(BaseModel):
    video_id: str
    start_s: float
    end_s: float
    modality: str
    score: float
    thumbnail_url: str | None = None
    preview: str | None = None
    raw_score: float | None = Field(default=None, description="Cosine score inside its own collection, before RRF")
    rank_in_modality: int | None = Field(default=None, description="Rank inside its own collection, before RRF")

Code walkthrough

• Every answer in Parts 1 and 2 comes with a list of these: the evidence behind it. This is the "source" part of a good answer.
• start_s and end_s are the "time" part: the eval harness checks them against the gold window.
• modality says which signal the evidence came from (visual or text in Part 1). Part 2 returns the same shape, which is why one harness can compare both.
• raw_score and rank_in_modality show where the item stood inside its own collection before fusion; Module 5 uses them to explain rank fusion.

What happens when someone asks c01

1. A user types "Which service did Tidewater remove in March?" into Part 1's UI. The UI sends it to POST /search with top_k 5.
2. Part 1 embeds the question twice (once for frames, once for transcript text) and searches two collections, 10 candidates each.
3. In the 27 Oct 2026 run, the best transcript chunk covered 0.0 to 23.0 s. It overlaps the gold window (17.333 to 36.133 s), so retrieval scored rank 1. The best frame was from 175 to 180 s: the cost chart, not the service map.
4. The two rankings are merged by rank. The model receives the transcript text of the text sources and only a time range for the frames (Module 6 diagnoses this failure).
5. The transcript says "the service in the red box ... removed in March". The name is not in any text the model received. The model answers that the name "is not mentioned".
6. The harness marks c01 as rank 1 and answer wrong: almost correct.

Design decisions

Cross-modal questions are the largest group because they are the lesson; visual and speech questions prove that each path works on its own, so a cross-modal failure cannot be blamed on a dead path.

A string check is crude, and that is a feature. It cannot be talked round by a fluent answer, and it costs nothing to run. Its limit is the reverse: an answer that names "Label Printer" and then says something false about it still passes. The course states that limit wherever it quotes answer rates.

When to choose differently

What can go wrong

A keyword match finds the wrong segment. "Geofence" is spoken in segment 3 and written in 6 and 12; "Tracking Service" is drawn on four slides. A system that ranks by shared words will often return the wrong one of these. The defence is to rank by meaning and to test with the traps in place: c02's note says "'Geofence Worker' is also written on the Next steps slide, but only this window says 'next quarter'". In the comparison run Part 1 missed c02 entirely (no source in the gold window); Part 2 ranked it first.

A near-duplicate wins. A frame of the wrong service map looks almost the same as the right one. The gold windows for c01 and c02 are different segments, so picking the wrong map scores as a miss. Module 3 shows a real case: for c03, CLIP's best frame came from the Week 42 chart, not the Week 37 one.

The distractor leaks into answers. The pizza segment has temperatures, timings and a "Friday". For v01, Part 1's top source was a pizza-slide frame at 125 to 130 s, and the answer was "I could not find which meeting room the architecture review is held in from the video evidence." The retrieval failed; the answer at least said so.

The answer sounds right and is not. This is c01 in Part 1, and it is the failure the course is built around. No exception is raised, no log line is written, the source is in the right place. Only a check against a known answer catches it.

Changing the voice changes the timeline. A different voice speaks at a different speed, so every segment moves. The build regenerates segments.json and questions.jsonl together from the measured timeline; re-run the evals afterwards.

Fixed: the original could not answer at all. The original v1 hard-coded gemini-2.5-flash-lite, which now returns 404 "no longer available to new users". This was found during the integration run with the author's key. The hardened code reads the model from the setting GENERATION_MODEL (default gemini-3.5-flash-lite), keeps one client with a timeout, and retries on 429s. Tests: test_health_metrics_and_errors checks that /health reports gemini-3.5-flash-lite, and test_generator_context_and_fallbacks checks that a missing key or a failed call returns the raw evidence instead of an error.

Known limits. The test video is one synthetic review with a synthetic voice, still slides and no camera motion. Its 26 questions give no confidence intervals. The answer check accepts any answer that contains the string. The video's world (Tidewater Freight, Priya Nair, the services) is fictional and exists only in segments.json.

Try it yourself

cd source-code/samples
python3 -c "import json; s=json.load(open('segments.json')); [print(g['index'], g['key'], g['start_s'], g['end_s']) for g in s['segments']]"

What you should see

Twelve lines, from 1 title 0.0 17.333 to 12 next 190.0 202.467. Line 2 is 2 arch_removed 17.333 36.133, c01's gold window.

cd source-code/samples
python3 -c "import json, collections; print(collections.Counter(json.loads(l)['kind'] for l in open('questions.jsonl')))"

What you should see

Counter({'cross_modal': 8, 'visual': 7, 'speech': 7, 'distractor': 4}).

cd source-code/samples
uv run build_test_video.py --tts say --out-dir /tmp/iok-video

What you should see

On a Mac with ffmpeg installed: a line slides: 12 rendered, self-check passed, a narration line saying say was used, then the video, segments and question files written to /tmp/iok-video. The committed files in samples/ are not touched. If your Mac's Samantha voice differs, the timings differ slightly; that is expected.

In the Part 1 UI (make up-part1, then open http://localhost:8511): upload samples/test_video.mp4 on Upload and index, then on Ask type "Which service did Tidewater remove in March?". Read the answer, then the sources and their previews. Whatever the transcript says about that moment, it speaks of "the service in the red box" and never names it, because the name is never spoken.

Check yourself

1. Why is c01 a cross-modal question and v01 a visual one?

v01's answer (Harbor-4) is drawn on screen and never spoken, so the picture alone answers it. c01 needs the name, which is only on screen, and "removed in March", which is only spoken. Neither signal alone contains the answer.

1. A system scores 100% on visual and speech questions and 0% on cross-modal ones. What does that tell you?

Both paths work on their own: it can read the screen and it can read the speech. What fails is connecting a spoken pointer ("the red box") to the on-screen value at the same moment, a binding problem rather than a broken component.

1. Why does the test video contain two almost identical service maps?

So that matching "a service map" is not enough. c01 and c02 have different gold segments; only a system that uses what was said ("removed in March" or "adding next quarter") can pick the right one. Without the pair, a lucky frame match would pass.

1. Part 1's c01 answer cites evidence and its top source is in the gold window. Why does it fail?

It does not contain "Label Printer". A good answer needs the name or value, a time and a source; this one has a source and a reason but not the name, because the name never reached the model as text.

1. Why does the answer check use short strings like "Label Printer" instead of a model as a judge?

It is cheap, repeatable and cannot be persuaded by fluent wording. Its limit is that it passes any answer containing the string, even one that adds a false claim.

Key terms