Build an agent from scratch

Goal

Build a working coding/file agent against a raw LLM API with no agent framework: the loop, the tools, the safety gates, and the evaluation harness, all yours. The point is demystification — after this, every framework is recognizably a while-loop with opinions, and you know exactly where the reliability and security engineering has to live.

Subject: full brief & instructions

Practices

• Agents, RAG & tool use — the agent loop, tool design, injection defenses, and evaluation: the whole concept, hand-built.
• LLM fundamentals for engineers — context-window budgeting, structured output, and token economics, felt from the driver's seat.
• API design — tool definitions are API design for a model audience.
• Threat modeling — model+tools as a trust boundary.

Milestones

1. Chat loop. A terminal chat against the raw API with streaming and a system prompt kept as a versioned file. No tools yet — just the conversation state you now own.
2. One tool. Define read_file with a tight JSON schema, validate the model's call, execute it, return the result as context. One full function-calling round trip, understood end to end.
3. The loop. Let the model chain tool calls until it declares done — with bounded iterations and a visible trace of each step. Congratulations: it's an agent.
4. Tools with teeth. Add list_files, search, and edit_file — the last behind a human-approval gate. Treat file contents as untrusted input: try to prompt-inject your own agent via a file it reads, then decide what that means for your design.
5. Observability. Log every run — inputs, retrievals, tool calls, tokens, cost — to a trace you can replay. Debug one bad run from the trace alone.
6. Evals. Build a golden set of ~10 tasks with checkable outcomes and score the agent on it; change the system prompt and measure whether it helped.

Stretch goals

• Context compaction: summarize old turns when the window fills, without losing the plot.
• A planner/worker split — observe what context partitioning buys and costs.
• Wrap the tool surface as an MCP server (spec) so other clients can use it.

Related

• Agents, RAG & tool use and LLM fundamentals for engineers — the concept notes whose # Practice cites this.
• Build an agent from scratch — subject — the full spec and acceptance checks.
• RAG over the knowledge base — the sibling exercise; bolt its retriever on as a tool for agentic RAG.
• Alfred itself — this repository is a production instance of the pattern; compare designs.