Overview
Part 1, The Architecture That Should Work. A video question-answering system stitched together from separate models: CLIP looks at frames, Whisper turns speech into text, MiniLM embeds that text, two vector collections are searched and their rankings merged. It answers "How long are raw GPS pings kept?" correctly. Ask "Which service did they remove in March?" and it finds the right moment yet answers that the service "is not mentioned", because the name is on screen and the "removed in March" is spoken, and the two never meet. The problem is not the prompt, the chunking or the vector database. It is the architecture. Part 2, The Redesign. Fewer components and a different retrieval philosophy: each window of video and its audio become one vector from one multimodal embedding model, in one collection, with no fusion. One decision removes a whole category of bugs. On the same 26 questions, cross-modal answers go from 0 of 8 to 8 of 8, and you will see exactly what that number does and does not prove. Part 3, StyleMe. A fashion shopping agent over text, photos and multi-turn chat. Before any retrieval it decides what the shopper wants: a greeting, a clarifying question (never more than three) or a search. Then the work that makes it a product: grounded replies that never name an item it did not find, server-side sessions, guards on uploads and traffic, tracing, an evaluation suite with gates, containers, and a Cloud Run deployment with keyless CI/CD. Who it is for: anyone who knows a little Python and wants to understand, rebuild or extend a multimodal retrieval system and a production agent, from beginners to working engineers. Each module says what to read first.
What you will learn
- 26 written parts, yours for good
- 9h 33m of reading, measured not estimated
- Written for the intermediate level
- Every future revision included
Curriculum
26 sections · 26 parts · 9h 33m46 min free to read · 8h 47m with the project
01Start here1 part · 3 min
02Module 0: Project overview: a guided tour of all three parts1 part · 19 min
03Module 1: The problem with multimodal search1 part · 24 min
- Module 1: The problem with multimodal searchPreview24 min
04Module 2: The stitched architecture1 part · 23 min
- Module 2: The stitched architecture23 min
05Module 3: Frames and CLIP1 part · 23 min
- Module 3: Frames and CLIP23 min
06Module 4: Speech: Whisper and MiniLM1 part · 22 min
- Module 4: Speech: Whisper and MiniLM22 min
07Module 5: Two collections and rank fusion1 part · 19 min
- Module 5: Two collections and rank fusion19 min
08Module 6: "Almost correct": diagnosing the failure1 part · 22 min
- Module 6: "Almost correct": diagnosing the failure22 min
09Module 7: One embedding space with Gemini Embedding 21 part · 20 min
- Module 7: One embedding space with Gemini Embedding 220 min
10Module 8: Windows, overlap and the audio surprise1 part · 23 min
- Module 8: Windows, overlap and the audio surprise23 min
11Module 9: One collection, no fusion, grounded answers1 part · 22 min
- Module 9: One collection, no fusion, grounded answers22 min
12Module 10: Measuring v1 against v21 part · 22 min
- Module 10: Measuring v1 against v222 min
13Module 11: StyleMe: the product problem1 part · 23 min
- Module 11: StyleMe: the product problem23 min
14Module 12: High-level design1 part · 22 min
- Module 12: High-level design22 min
15Module 13: Low-level design: API, sessions and data1 part · 24 min
- Module 13: Low-level design: API, sessions and data24 min
16Module 14: The planner: intent before retrieval1 part · 23 min
- Module 14: The planner: intent before retrieval23 min
17Module 15: Clarification policy and the cap1 part · 24 min
- Module 15: Clarification policy and the cap24 min
18Module 16: Multimodal product search and ingestion1 part · 24 min
- Module 16: Multimodal product search and ingestion24 min
19Module 17: Grounded generation1 part · 26 min
- Module 17: Grounded generation26 min
20Module 18: The frontend1 part · 25 min
- Module 18: The frontend25 min
21Module 19: Tracing with Langfuse and OpenTelemetry1 part · 21 min
- Module 19: Tracing with Langfuse and OpenTelemetry21 min
22Module 20: Evaluating the agent1 part · 24 min
- Module 20: Evaluating the agent24 min
23Module 21: Security: threats, guards and limits1 part · 24 min
- Module 21: Security: threats, guards and limits24 min
24Module 22: Containers and the local stack1 part · 24 min
- Module 22: Containers and the local stack24 min
25Module 23: Deploying StyleMe with CI/CD1 part · 24 min
- Module 23: Deploying StyleMe with CI/CD24 min
26Module 24: What changes in production1 part · 23 min
- Module 24: What changes in production23 min
Most AI projects stop when the answer looks correct. This course starts there. It is one course in three parts, built from real code and measured at every step.
Part 1, The Architecture That Should Work. A video question-answering system stitched together from separate models: CLIP looks at frames, Whisper turns speech into text, MiniLM embeds that text, two vector collections are searched and their rankings merged. It answers "How long are raw GPS pings kept?" correctly. Ask "Which service did they remove in March?" and it finds the right moment yet answers that the service "is not mentioned", because the name is on screen and the "removed in March" is spoken, and the two never meet. The problem is not the prompt, the chunking or the vector database. It is the architecture.
Part 2, The Redesign. Fewer components and a different retrieval philosophy: each window of video and its audio become one vector from one multimodal embedding model, in one collection, with no fusion. One decision removes a whole category of bugs. On the same 26 questions, cross-modal answers go from 0 of 8 to 8 of 8, and you will see exactly what that number does and does not prove.
Part 3, StyleMe. A fashion shopping agent over text, photos and multi-turn chat. Before any retrieval it decides what the shopper wants: a greeting, a clarifying question (never more than three) or a search. Then the work that makes it a product: grounded replies that never name an item it did not find, server-side sessions, guards on uploads and traffic, tracing, an evaluation suite with gates, containers, and a Cloud Run deployment with keyless CI/CD.
Source code
The complete, runnable project as one zip: the code the modules explain, with its tests and the files to run and deploy it.
The source code download comes with buying this project.
Submit your build
When the last milestone runs, hand in the repository and three lines about it. That is what the project page will show for you, and what you can send with a job application.
Sign in to submit your build: the repository link and three lines about it.
Reviews
to review this project once you have finished it.