CourseAI System Design: The Complete Course · Module 5: Data Architecture · part 38 of 107
Part 38 · Module 5: Data Architecture

Lab: Intake, from source listing to searchable chunks

126 min read·
Module 5 · Lab · About 60 minutes · Needs: lessons 1 to 3 of this module, and the Module 4 lab (or nothing: Step 1 rebuilds Modules 1 to 4)

What you are building

CODE TOUR One sync, three stores sources.py 471 listed governance.py admit() intake.py sync() object store redacted text operational the record vector store derived copy set aside 15 with reasons test_ingest.py pins the behaviour ADR-002, design step 5 the decision, the figure
The lab's piece: the listing (grey) goes through the checks and one sync (blue) into three stores; 15 documents are set aside with reasons (red), and tests pin the behaviour (green).

Three lessons built the data path in pieces: Lesson 1 decided what changed, Lesson 2 where each copy lives, Lesson 3 what a document must pass first. Leila wants to see one document go through all of it, and Gareth wants the run repeatable. This lab puts the lessons' 15 files into the repository unchanged and adds four of its own: northdesk/sources.py, the listing each source hands over; northdesk/governance.py, Lesson 3's checks in one order; northdesk/intake.py, one sync from listing to vectors; and northdesk/datapath.py, the data path figure for docs/design.md. tests/test_ingest.py pins the behaviour, and docs/adr/ADR-002-data-path.md records the decision.

No model is called and no key is needed. Every number comes from the shared data.

Open a new Colab notebook, call it module-05-lab.ipynb, and run each cell in order.

The rest of this course is yours to keep

This course is bought on its own, once, and stays readable afterwards, including the parts added to it later.

Buy this course · ₹399See what is inside

Or open everything with a plan