RAG over the knowledge base

Goal

Build a question-answering RAG pipeline whose corpus is this knowledge base — a corpus you know intimately, which is precisely what makes retrieval failures visible. The skill proved: every stage of the RAG pipeline, plus the discipline of evaluating retrieval separately from generation.

Subject: full brief & instructions

Practices

• Agents, RAG & tool use — the RAG pipeline and its "retrieval is the bottleneck" law, tested empirically.
• LLM fundamentals for engineers — embeddings as the second primitive; context budgeting when stuffing chunks.
• Search systems — lexical vs. vector vs. hybrid retrieval, in miniature.

Milestones

1. Corpus & chunking. Walk tech-kb/, parse frontmatter, and chunk structure-aware (by heading section, metadata attached: theme, title, tags, path). Compare against naive fixed-size chunks on a few files — see why structure wins.
2. Embed & index. Embed every chunk, store vectors + metadata locally (SQLite or a small vector store). Rebuildable from scratch with one command.
3. Retrieval + its own eval. Top-k vector search; then add lexical (BM25-ish) and combine. Build a ~20-question eval set mapping questions to the note that answers them, and score recall@k for vector, lexical, and hybrid. This milestone is the exercise's heart.
4. Cited generation. Stuff retrieved chunks into the prompt with source paths; answers must cite the notes they used. De-duplicate and budget the context.
5. End-to-end evals. Golden Q&A pairs, LLM-as-judge for faithfulness ("is the answer supported by the cited chunks?"), spot-check the judge. Now change the chunking and measure what moves.

Stretch goals

• Agentic RAG: expose retrieval as a tool to the agent from scratch and let it decide when to search.
• Incremental re-indexing triggered by git commits touching tech-kb/.
• A reranker stage; measure whether it earns its latency on a corpus this small.

Related

• Agents, RAG & tool use and LLM fundamentals for engineers — the concept notes whose # Practice cites this.
• RAG over the knowledge base — subject — the full work statement and acceptance checks.
• Search systems — the retrieval half, at production scale.
• Build an agent from scratch — the sibling exercise this one plugs into.