ProjectBuild a Permission-Aware Knowledge Hub as an MCP Server: the explanation · Module 17: Evaluating retrieval and the 0-leak gate · part 19 of 22
Part 19 · Module 17: Evaluating retrieval and the 0-leak gate
Module 17: Evaluating retrieval and the 0-leak gate
24 min read·
About 55 minutes · Needs: Module 6 (the seed company and its traps), Module 9 (hybrid search and ranking), Module 10 (permission filtering before ranking) · Code: server/evals/run.py, server/evals/reports/, seed/eval/questions.jsonl, seed/eval/permissions.jsonl, seed/tools/build_eval.py
Where this fits
This module (blue) is the evaluation box: it asks the seed company's golden questions through the real search and access code and turns the answers into numbers a gate can check.
Unit tests check that one function does what its author meant. An evaluation asks the whole retrieval path many realistic questions and scores the answers against rules written in advance. This module opens the evaluation box: the two eval files, the two suites, every metric with its formula, the real results of 25 October 2026, and the gate that can stop a release.
The rest of this project is yours to keep
This project is bought on its own, once, and stays readable afterwards, including the parts added to it later.