Topic 3: The Decision That Comes First

The most important idea in this module: RAG is one tool among several. Choose the architecture before writing code. No amount of chunking tuning fixes the wrong architecture.

3.1 RAG vs Long Context

Intuition: Long context is reading the whole book for every question. RAG is using the index to jump to the right page.

Prompt caching lets providers reuse a repeated prompt prefix at a lower price and latency. If many users query the same fixed documents, long context plus caching can compete with RAG. It helps less when documents differ per user or traffic is sparse. Pricing varies by provider, so benchmark.

Use cases

3.2 RAG vs Fine-Tuning

Intuition: RAG is giving an employee a reference manual. Fine-tuning is sending them to a training course. Courses teach behaviour, not next week's facts.

Fine-tuning is a poor way to add facts: they're fuzzy, need retraining to update, and can't be cited. Try in this order: prompting → RAG → fine-tuning. You can also combine them: fine-tune for style, use RAG for facts.

Use cases

3.3 RAG vs Tool Calling: Structured vs Unstructured Data

Intuition: You don't search a library for your bank balance. You ask the bank.

Tool calling: the LLM asks your code to run a function (like get_order_status("4512")), your code runs it, and the result goes back to the model.

Why vector search fails on structured questions: "How many orders over $100 shipped last week?" needs filtering and counting. Similarity search returns text that looks similar, not an exact number.

Use cases

Watch out: if an LLM writes SQL, give it a read-only user and an allow-list of tables.

3.4 RAG vs Agentic Search

Intuition: RAG is your own filing cabinet. Agentic search is a research assistant who goes out, searches, reads, decides what to search next, and reports back.

Use cases

Watch out: always cap the number of agent steps, and treat text from external pages as data, never as instructions.

3.5 Hybrid Strategies and the Three Deciding Variables

Intuition: A hospital triage desk routes each patient to the right specialist. Production assistants route each question the same way.

Three variables decide the architecture for each data source:

Tip: if 80% of questions are the same 20, human-reviewed answers for those plus RAG for the rest is often cheaper and more accurate. A university chatbot did exactly this for fees, deadlines, and hostels.

ShopSphere, decided:

"Can I return my blender from order 4512?" needs both the return policy (RAG) and the delivery date (tool call). A router sends it down both paths, and the LLM combines the results. You'll build exactly this in Topic 6.