rgoussu.dev
rgoussu@goussu: ~/library/ai-engineering
❯ ~/library/ai-engineering cat agents-rag-and-tool-use.md

Agents, RAG & tool use

# The systems layer over LLMs — grounding answers in retrieved data, giving models tools, and the loop that turns a model into an agent.

Conceptsaved 2026-08-08updated 2026-08-09 #agents#rag#tool-use#llm#ai-engineering

Overview

Raw models complete text; useful systems ground them in your data (RAG), let them act through your functions (tool use), and loop them toward goals (agents). All three are plain software engineering wrapped around a stochastic core — retrieval pipelines, schema-validated function calls, a while-loop with an exit condition — and they inherit software engineering's obligations: evaluation, observability, and security at the trust boundaries. (This very repository — Alfred — is one of these systems.)

Key points

  • RAG in one line: retrieve relevant context, stuff it into the prompt, cite it — grounding generation in data the model wasn't trained on and keeping answers current and attributable. The pipeline: chunk documents (structure-aware beats fixed-size) → embed → index → retrieve (hybrid lexical+vector with reranking beats either alone — see search systems) → generate with citations. Retrieval quality is almost always the bottleneck — debug the retriever before blaming the model.

    flowchart LR
    DOC[Documents] --> CH["Chunk (structure-aware)"]
    CH --> EM[Embed]
    EM --> IX[Index]
    IX --> RE["Retrieve (hybrid lexical + vector, reranked)"]
    RE --> GEN["Generate with citations"]

  • Tool use: the model emits a structured function call, your code executes it, results return as context, generation continues. Tool definitions are API design for a model audience (API design): clear names, tight schemas, good error messages — the model reads them like documentation, because it is.

  • An agent is a loop: model → tool call(s) → observe results → decide next step → repeat until done. Reliability engineering, not magic: bounded iterations, checkpoints, idempotent tools where possible, human-approval gates on irreversible actions, and state kept outside the context window (files, task lists) for anything long-running.

    flowchart TB
    M[Model] --> TC["Tool call(s)"]
    TC --> OB[Observe results]
    OB --> DE[Decide next step]
    DE -->|repeat| M
    DE -->|"done (bounded iterations)"| EX[Exit]

  • Context engineering is the core skill: what enters the window, in what order, and what gets summarized away — system prompts as versioned artifacts, retrieved chunks budgeted, conversation history compacted. Multi-agent patterns (planner/worker, fan-out) are context-partitioning strategies before they are anything else.

  • Security: prompt injection is the problem: retrieved documents, web pages, and tool outputs are untrusted input that the model may follow as instructions. Treat model+tools as a trust boundary (threat modeling): least-privilege tools, output validation, no secrets in prompts, human gates where it matters. Unsolved in the general case — design assuming it happens.

  • Evaluation or superstition: golden datasets, LLM-as-judge with spot-checked rubrics, tracing every run (inputs, retrievals, tool calls, tokens) — regression-test prompts and pipelines like code, because they are.

  • To explore: MCP as the emerging tool-integration standard, agentic RAG (retrieval as a tool), memory architectures, multi-agent orchestration costs.

Practice

  • Gandalf prompt-injection game (source) — extract the password level by level; half an hour of attacking makes "design assuming injection happens" visceral in a way reading never will.
  • Write an MCP server (source) — expose a small tool surface (this repo's notes are a fine target) over the emerging standard; tool definitions as API design, practiced for real.
  • Build an agent from scratch (source) — the loop, tools, and eval harness with a raw LLM API and no framework, until the magic is demonstrably a while-loop.
  • RAG over the knowledge base (source) — chunk → embed → hybrid retrieve → cite, over this very repo, with retrieval evals to prove where the bottleneck actually is.

Related