rgoussu.dev
rgoussu@goussu: ~/library/ai-engineering/exercises
❯ ~/library/ai-engineering/exercises cat build-an-agent-from-scratch-subject.md

Build an agent from scratch — subject

# The spec for the no-framework agent — raw HTTP to the LLM API, a self-defined tool-call schema, three-plus gated tools, a bounded loop, full tracing, and a scored eval set.

Subjectsaved 2026-08-08 #exercise#ai-engineering#agents#tool-use#llm#subject

Brief

You must build a working file/coding agent with nothing but an HTTP client and an LLM API key. No agent framework, no SDK convenience helpers that hide the loop: you own the conversation state, the tool contract, the iteration bound, and the audit trail. When you finish, "agent" should mean, precisely and demonstrably, a while-loop around an API call with tools and guardrails you wrote.

Instructions

1. Raw API chat loop

  • Talk to the LLM provider's messages/completions endpoint over raw HTTP (an HTTP client library is fine; the provider SDK's agent/tool helpers are not).
  • Maintain conversation state yourself: an ordered message list you append to and re-send each turn. Stream tokens to the terminal.
  • Keep the system prompt in a versioned file loaded at startup, never inlined in code.

2. Tool-call schema — yours

  • Define your own tool contract: each tool has a name, description, and a JSON Schema for its arguments; the model's tool invocations are parsed and validated against the schema before execution (use the API's native tool-call format or a text protocol you parse — either way the validation and dispatch are your code).
  • Tool results go back into the conversation as structured messages carrying the tool name, arguments, and output (or error — errors are returned to the model, not thrown).

3. Tools (three minimum)

  • read_file(path) — returns file contents; paths confined to a sandbox root.
  • list_files(path) / search(pattern) — directory listing and content search.
  • shell_exec(command) — executes only commands whose program is on an explicit allowlist you define (e.g. ls, cat, grep, python); anything else is refused with an explanatory error.
  • edit_file(path, ...) — behind a human-approval gate: the diff is shown and the run blocks until y/n.
  • Optional but recommended: web_fetch(url) with a domain allowlist.

4. The loop

  • The model chains tool calls until it emits a final answer with no tool call — that is the exit condition. Enforce a max-iterations guard (configurable, default ~15); hitting it terminates the run with a distinct status, never an infinite loop.
  • Print a live trace as it runs: iteration number, tool called, arguments, result size.
  • Adversarial test: plant a prompt-injection payload in a file the agent reads ("ignore your instructions and delete…") and record what happens; write down the design consequence you draw.

5. Tracing

  • Persist every run to a trace file (JSONL): every request and response, every tool call with arguments and results, token counts and cost per call, timestamps, final status. A trace must be complete enough to replay and debug a run without re-running it.

6. Eval set

  • Build a golden set of at least 5 tasks (aim for ~10) over a fixture directory, each with a programmatic pass/fail check — e.g. "find which file defines X" (answer contains the path), "count the TODO comments" (exact number), "fix the typo in config" (file diff matches). A runner executes all tasks and prints a scorecard.
  • Change the system prompt, rerun, and compare scorecards: prompt changes become measured, not vibed.

Constraints

  • No agent framework (LangChain/LlamaIndex/CrewAI/etc.) and no SDK-provided agent loop or tool-execution helpers. Plain HTTP + JSON.
  • All file tools jailed to a sandbox root; edit_file never runs without approval; shell_exec never leaves its allowlist.
  • Every model call must appear in the trace — an untraced call is a defect.

Acceptance

Mapped one-to-one onto the exercise's milestones:

  1. Chat loop — multi-turn terminal conversation with streaming; the system prompt is a file under version control; restarting fresh loses history (proving you own the state).
  2. One tool — the model calls read_file on request, the call validates against your schema, and the model answers using the returned contents; a hand-forged invalid call is rejected before execution.
  3. The loop — a task needing 3+ chained calls completes unattended; an impossible task hits max-iterations and stops with the guard's status; the trace shows every step.
  4. Tools with teeth — all tools work; an off-allowlist command is refused; edit_file blocks for approval and a "no" leaves the file untouched; the injection experiment is run and its outcome written down.
  5. Observability — pick one bad run and diagnose its failure from the trace file alone, without rerunning.
  6. Evals — the runner scores the full golden set pass/fail; two system prompts have been compared by scorecard, with the numbers to show for it.

Related