rgoussu.dev
rgoussu@goussu: ~/library/ai-engineering/exercises
❯ ~/library/ai-engineering/exercises cat rag-over-the-knowledge-base-subject.md

RAG over the knowledge base — subject

# The work statement for the RAG build over tech-kb — structure-aware chunking rules, hybrid retrieval with rank fusion, path-citing generation, and a hit-rate-scored retrieval eval set.

Subjectsaved 2026-08-08 #exercise#ai-engineering#rag#embeddings#retrieval#evaluation#subject

Brief

Your corpus is this repository's tech-kb/ — an OKF bundle of markdown notes with YAML frontmatter, heading-structured bodies, and an index hierarchy you know by heart. Build a question-answering pipeline over it: ask "what does the KB say about lock-free concurrency?" and get an answer that cites the exact files it drew from. Because you know the corpus intimately, every retrieval failure is visible — which is the point.

Instructions

1. Corpus & structure-aware chunking

  • Walk tech-kb/ recursively; skip reserved files (index.md, log.md, AGENTS.md, AGENTS.md) and anything with visibility: private.
  • Parse each note's YAML frontmatter; carry title, type, tags, theme, and the bundle-relative path as metadata on every chunk from that file.
  • Chunk by heading section: each #/## section is a chunk (split oversized sections at paragraph boundaries; merge tiny trailing ones), and each chunk records its heading path (e.g. Deep dive Java > # Key points). Frontmatter is metadata, never chunk text.
  • Build the naive comparator too — fixed-size character chunks with overlap — and diff the two chunkings over a handful of files to see what structure buys.

2. Embed & index

  • Embed every chunk; store vectors + chunk text + metadata locally (SQLite, or a small embedded vector store). One command rebuilds the whole index from a clean checkout.

3. Hybrid retrieval + retrieval eval (the heart)

  • Vector: top-k by cosine similarity over the embeddings.
  • Lexical: BM25(-ish) keyword scoring over chunk text (SQLite FTS5 or a small BM25 implementation).
  • Fusion: combine both rankings with reciprocal rank fusion (RRF) or weighted score blending into one hybrid ranking.
  • Eval set: write at least 10 (target ~20) question → expected-source-file pairs, mixing phrasing that mirrors note wording with paraphrases that don't. The eval runner scores hit rate (recall@k) at k = 1, 3, 5 for vector-only, lexical-only, and hybrid, and prints a comparison table plus the misses.

4. Cited generation

  • Stuff the top chunks into a generation prompt with their source paths attached; de-duplicate chunks from the same section and enforce a context-token budget.
  • The prompt requires answers to quote the bundle paths they used (e.g. "per /java/java-deep-dive.md …"); an answer with no file citation is a failed answer. Refuse gracefully when retrieval returns nothing relevant.

5. End-to-end evals

  • Golden Q&A pairs over the corpus; an LLM-as-judge scores faithfulness ("is every claim supported by the cited chunks?"); spot-check the judge on a sample by hand before trusting it.
  • Then change one variable — chunking strategy (structure-aware vs. naive) — rerun both eval layers, and report what moved: retrieval hit rate, faithfulness, or both.

Constraints

  • Retrieval is evaluated separately from and before generation — no skipping to end-to-end vibes.
  • Local storage only (no hosted vector DB); the index is disposable and rebuildable.
  • Answers must cite bundle-relative file paths verbatim; citations invented by the model (paths not among the retrieved chunks) count as failures.

Acceptance

Mapped one-to-one onto the exercise's milestones:

  1. Corpus & chunking — the chunker emits heading-aligned chunks with full metadata (path, theme, title, tags, heading path); private and reserved files are absent; the structure-vs-naive comparison exists in writing with examples.
  2. Embed & index — one command builds the index from scratch; chunk count and per-theme distribution are reported; a rebuild is deterministic modulo embedding API.
  3. Retrieval + eval — the eval runner prints recall@1/3/5 for vector, lexical, and hybrid on the 10+ question set; hybrid ≥ the better single method on the set, or you can explain why not.
  4. Cited generation — answers quote real retrieved file paths; the no-relevant-hit case degrades to an honest "not in the KB"; context stays within the token budget.
  5. End-to-end evals — faithfulness scores exist for both chunking strategies, the judge has been spot-checked, and the chunking change's effect is stated with numbers.

Related