Intent Over Keywords: From Stitched Embeddings to a Production Fashion Agent
Most AI projects stop when the answer looks correct. This course starts there. It is one course in three parts, built from real code and measured at every step.
Part 1, The Architecture That Should Work. A video question-answering system stitched together from separate models: CLIP looks at frames, Whisper turns speech into text, MiniLM embeds that text, two vector collections are searched and their rankings merged. It answers "How long are raw GPS pings kept?" correctly. Ask "Which service did they remove in March?" and it finds the right moment yet answers that the service "is not mentioned", because the name is on screen and the "removed in March" is spoken, and the two never meet. The problem is not the prompt, the chunking or the vector database. It is the architecture.
Part 2, The Redesign. Fewer components and a different retrieval philosophy: each window of video and its audio become one vector from one multimodal embedding model, in one collection, with no fusion. One decision removes a whole category of bugs. On the same 26 questions, cross-modal answers go from 0 of 8 to 8 of 8, and you will see exactly what that number does and does not prove.
Part 3, StyleMe. A fashion shopping agent over text, photos and multi-turn chat. Before any retrieval it decides what the shopper wants: a greeting, a clarifying question (never more than three) or a search. Then the work that makes it a product: grounded replies that never name an item it did not find, server-side sessions, guards on uploads and traffic, tracing, an evaluation suite with gates, containers, and a Cloud Run deployment with keyless CI/CD.
Who it is for: anyone who knows a little Python and wants to understand, rebuild or extend a multimodal retrieval system and a production agent, from beginners to working engineers. Each module says what to read first.
How each module is laid out
| Section | Answers |
|---|---|
| Where this fits | the same map of all three parts in every module, with this module's pieces highlighted |
| What it does | the idea in one picture, then each production practice at a glance: what, why, how, when, where |
| Use cases and examples | the same practice in this project and in other industries |
| How it works | the mechanism step by step, one figure per step, with real values |
| Code tour | the real files (and, where it teaches something, the original code next to the improved one), short excerpts, every line explained, concept boxes for new ideas, and a "what happens when" trace |
| Design decisions | what was chosen, the alternatives and why, with measured numbers, and when to choose differently |
| What can go wrong | failures and attacks, how the code handles each, and the test that proves it |
| Try it yourself | exact commands and what you should see |
| Check yourself, Key terms | a short quiz and a glossary |
Figures use one colour language: blue is our system, amber is a model (CLIP, Whisper, MiniLM, Gemini) or anything uncertain, green is correct or safe, red is failure or attack, grey is background.
Modules
| # | Module | Main code |
|---|---|---|
| 0 | Project overview: a guided tour of all three parts | screenshots of the real apps, no code |
| Part 1: The Architecture That Should Work | ||
| 1 | The problem with multimodal search | samples/, originals/v1/ |
| 2 | The stitched architecture | part1-video-rag-stitched/src/ |
| 3 | Frames and CLIP | extractors/media.py, embedders/local_models.py |
| 4 | Speech: Whisper and MiniLM | embedders/local_models.py, text/chunking.py |
| 5 | Two collections and rank fusion | vectorstore/qdrant_store.py, retrieval/ |
| 6 | "Almost correct": diagnosing the failure | samples/build_test_video.py, generation/generator.py |
| Part 2: The Redesign | ||
| 7 | One embedding space: Gemini Embedding 2 | part2-video-rag-unified/src/embedders/gemini_embedder.py |
| 8 | Windows, overlap and the audio surprise | extractors/media.py, ingestion/pipeline.py |
| 9 | One collection, no fusion, grounded answers | retrieval/retriever.py, generation/generator.py |
| 10 | Measuring v1 against v2 | evals/video/run.py, evals/video/reports/ |
| Part 3: StyleMe | ||
| 11 | StyleMe: the product problem | originals/styleme/, part3-styleme/backend/data/ |
| 12 | High-level design | api.py, agent/graph.py, docker-compose.yml |
| 13 | Low-level design: API, sessions, data and catalogue | api.py, sessions.py, catalog.py |
| 14 | The planner: intent before retrieval | agent/planner.py, agent/parsing.py, agent/prompts.py |
| 15 | Clarification policy and the cap | agent/planner.py, sessions.py, frontend/src/lib/clarify.ts |
| 16 | Multimodal product search and ingestion | agent/implementer.py, ingest.py, vectorstore.py |
| 17 | Grounded generation | agent/generator.py, agent/grounding.py |
| 18 | The frontend | part3-styleme/frontend/src/ |
| 19 | Tracing with Langfuse and OpenTelemetry | tracing.py, tests/test_tracing.py |
| 20 | Evaluating the agent | evals/run_evals.py, evals/datasets/, evals/results/ |
| 21 | Security | uploads.py, ratelimit.py, api.py, the video parts' api/uploads.py |
| 22 | Containers and the local stack | the Dockerfiles, docker-compose.yml, nginx.conf.template, Makefile |
| 23 | Deploy and CI/CD | deploy/, .github/workflows/ |
| 24 | What changes in production | llm.py, config.py, deploy/README.md |