rgoussu.dev
rgoussu@goussu: ~/library/ai-engineering
❯ ~/library/ai-engineering cat llm-fundamentals.md

LLM fundamentals for engineers

# The working model of large language models — tokens, context, sampling, capabilities and failure modes — needed to build on them soundly.

Conceptsaved 2026-08-08 #llm#ai#tokens#prompting#ai-engineering

Overview

You can build well on LLMs without training one, but not without a correct working model of what they do: autoregressive next-token prediction over a learned distribution, conditioned on everything in a finite context window. That single sentence explains most observed behavior — the fluency, the hallucinations, the sensitivity to prompt wording, the cost model — and replaces the two failure modes of intuition: treating the model as a database, or as a person.

Key points

  • Tokens are the native unit: text is split into subword tokens (~4 chars / ~0.75 words in English); pricing, context limits, and latency all count tokens. Streaming exists because generation is genuinely one token at a time, each conditioned on all before it.
  • The context window is the model's entire working memory: nothing persists between calls — "memory" in products is engineering (retrieval, summaries) that refills the window each turn. Long contexts degrade subtly (middle-loss); what's in the window and where is a real design surface.
  • Sampling knobs: temperature (0 ≈ argmax-deterministic-ish, higher = more varied), top-p; deterministic-looking ≠ correct — it's still a sampled distribution. Structured output (JSON modes, schema-constrained decoding, tool calls) is how you get machine-readable answers reliably, not "please output JSON".
  • Capabilities vs. failure modes: strong at transformation, synthesis, code, and instruction-following; unreliable as a fact store — hallucination is the model doing exactly its job (plausible continuation) without grounding. Countermeasures are architectural: retrieval for facts, tools for arithmetic/lookup, verification steps for stakes — see agents, RAG & tool use.
  • The lifecycle behind the API: pretraining (next-token, internet scale) → instruction tuning → preference alignment (RLHF-family) — knowing this explains why models follow instructions at all, and why system prompts steer them.
  • Embeddings are the other primitive: text → vectors where semantic similarity ≈ distance; they power search, clustering, RAG retrieval, and classification, at a fraction of generation's cost.
  • Engineering economics: cost = input + output tokens (output usually pricier); prompt caching, model-size routing (small model first, escalate on need), and batching are the levers; evals — not vibes — are how you know a prompt or model change helped.
  • To explore: fine-tuning vs. prompting vs. RAG (decision triangle), quantization & local inference, multimodality, inference mechanics (KV cache — why long contexts cost).

Practice

  • Build a BPE tokenizer (source) — implement byte-pair encoding from scratch following minbpe; after this, "pricing counts tokens" and tokenizer weirdness (spaces, numbers, non-English) stop being abstract.
  • Neural Networks: Zero to Hero (source) — Karpathy's build-it-yourself series from backprop to a small GPT; the single best way to make "autoregressive next-token prediction" a thing you've implemented, not read.
  • Embeddings mini-search (source) — embed a folder of your own notes, implement cosine-similarity top-k by hand, and probe where semantic ≈ distance holds and breaks; the cheap primitive, made concrete.
  • RAG over the knowledge base (source) — the countermeasure to hallucination, built end-to-end on this very repo.

Related