Lab: The production RAG service and the gauntlet

Module 14 · Capstone lab · About 150 minutes · Needs: the labs of Modules 5, 6, 7, 8, 9, 11 and 12 (or the cells in Step 1), and Docker on your own computer for Step 5

What you are building

Partners and the public help centre get Hollin Answers on Monday 30 November, if the capstone gauntlet passes. This lab turns thirteen modules into one service, then tries to break it. hollin/api/ puts FastAPI in front of Module 12's Service; hollin/stores/pgvector_store.py moves the vectors into PostgreSQL; docker-compose.yml runs the two. The gauntlet turns the project page's promises into checks run on every change: Module 11's gate on both tiers, Module 12's access and leak checks, Module 9's attack replays, freshness after the 8 October batch, and provider failure. .github/workflows/ci.yml runs it on every pull request and every night.

Uber's engineers worked this way on Genie, their internal on-call copilot: experts wrote "a golden set of 100+ test queries", an LLM judge scored answers from 0 to 5, and "the automated evaluation reduced experiment evaluation time from weeks to minutes" (Chakraborty, Settles and Raghavendra, "Enhanced Agentic-RAG", Uber engineering blog, 29 May 2025, https://www.uber.com/blog/enhanced-agentic-rag/). A fixed set of checks, run on every change, is what makes change safe.

The model is called in Steps 4 and 6 only.