Nine terms and one big idea. This is everything you need before rung 1 — and deliberately nothing more. Every other term in the framework is introduced at the rung where you first hit the wall it solves.
The big idea — how a model's "memory" actually works, and why it is your problem at every rung — gets its own section below: the Context pillar. Do not skip it. It is the single most load-bearing piece of background in the framework.
LLM (large language model). The kind of AI system this framework is about: a model trained on enormous amounts of text that produces text in response to text. Chatbots, assistants, and agents are all interfaces wrapped around an LLM.
Model. The trained artifact itself — the thing that turns input text into output text. Different models have different sizes, strengths, and costs. When people say "the model got it wrong," they mean this component, not the app around it.
Inference. What happens when a model runs: it takes your input and generates output. Training happened months earlier, once, at enormous expense; inference happens every time you press enter. This distinction matters later, when you meet questions like "where does inference run — someone's cloud, or a machine you control?"
Prompt. Everything you send the model in a turn. Not just your question — the instructions, the examples, the pasted material, all of it. The prompt is the whole input, and the quality of the output tracks the quality of the prompt more closely than newcomers expect.
Token. The unit models actually read and write — chunks of a few characters, roughly three-quarters of a word in English. Tokens are how context sizes are measured and how usage is billed. You never handle tokens directly, but every limit you will ever hit is denominated in them.
Context window. The model's fixed-size working area, measured in tokens. Everything the model can consider in a turn — instructions, conversation so far, pasted documents — must fit inside it. The Context pillar is about what this really implies; the short version: the window is not a nice-to-have detail, it is the model's entire reality.
System prompt. Standing instructions the model receives before your message — invisible to you in most products. It is why the same underlying model behaves differently in different apps, and why an assistant "knows" its name and rules.
Hallucination. Confident, fluent, wrong output. Not a glitch — a structural property of how these models work. A model with no tools cannot check anything; it can only produce plausible text. Some of that plausible text is false, and it arrives wearing the same confident tone as the true parts.
Verification. The habit that answers hallucination: checking factual output against something outside the model before you rely on it or pass it on. Verification is the first judgment skill of the ladder, and it never goes away — at higher rungs it changes form (checking citations, reviewing an agent's work, reading audit logs), but it is the same muscle. The gate out of rung 1 is not prompting skill; it is that verification has become instinct.
The model is stateless and its attention rots — so your job, at every rung, is deciding what deserves to be on the desk.
That sentence is the anchor for everything here, and this section is the anchor for half the framework. Context management shows up at every single rung — it just wears different names (prompting, templates, retrieval, memory, compaction). Learn the pillar once and every later term becomes a variation you already understand.
The foundational reframe, and the one most worth internalizing: the model has no memory. None. Every turn, it wakes up with amnesia and re-reads everything — the instructions, the whole conversation so far, whatever documents are attached — and only then produces its answer. What looks like a continuous conversation is a stack of independent awakenings, each handed a transcript.
The practical consequence: the model's context window is its entire reality for that turn. If something is not in the window, the model does not know it — not "forgot it," never knew it. Every technique in this framework for making AI useful is, underneath, a technique for getting the right things into that window.
Think of the context window as a desk. It has a fixed size, measured in tokens. Everything the model works with must physically fit on the desk: your instructions, the conversation, the pasted articles, the retrieved documents. The naive intuition is that a bigger desk is simply better, and that the right move is to pile everything on it. Both halves of that intuition are wrong — here is why.
Performance degrades as context grows — well before the window is full. A Chroma technical report (Hong, Troynikov & Huber, July 2025) tested 18 frontier models — GPT-4.1, Claude Opus 4, and Gemini 2.5 among them — and found all of them degrade non-uniformly as input grows, sometimes by 30–50%, far below their advertised limits.
Strong at the beginning of the context, strong at the end, weak in the middle. Liu et al. (Stanford, TACL 2024) showed accuracy drops of 30%+ when the key fact sits mid-context. The model skims the middle like a bored reader — put what matters at the edges.
Worse than being ignored: similar-but-irrelevant content actively misleads. A document that almost answers the question pulls the model toward wrong answers more effectively than obvious junk would. This is the specific reason "just dump everything in" backfires.
Put the three effects together and you get a rule of thumb: the context size at which a model stays reliably accurate is far below the number on the marketing page. Models advertising multi-million-token windows tend to hold high accuracy in the low hundreds of thousands — think ~150–400K effective on a 2M-window model, task-dependent. Plan around the effective number, not the advertised one.
Context engineering is the discipline all of this implies: deliberately curating what goes on the desk — what gets included, what gets summarized, what gets left out, what sits at the edges. It is the successor discipline to "prompt engineering": as the work gets more serious, wordsmithing the request matters less and curating the desk matters more. You will meet this term formally at rung 2, because rung 2 is where you start doing it on purpose.
| Strategy | What it is | Where it appears |
|---|---|---|
| Truncation / eviction | When the conversation outgrows the desk, the oldest material silently falls off. This is why long chats "forget" their beginnings. | Felt at rung 1, understood here |
| RAG (retrieval-augmented generation) | Instead of dumping every document on the desk, retrieve and place only the relevant ones, per question. | Rung 3 |
| Summarization / compaction | Replace bulky history with a dense summary to reclaim desk space. What well-built agents do continuously. | Rung 4 |
| Persistent memory | Storage across turns and sessions, outside the window entirely. Memory is across windows; context is within one. | Rung 4 |
One misconception to retire now: long context windows and RAG are complements, not competitors. A bigger desk does not remove the need to choose what goes on it; it changes the budget, not the job.
Attention cost is quadratic — doubling the context roughly quadruples the work. The model also builds a working structure called the KV cache that grows with context length and lives in GPU memory, which is why long contexts are slow and hardware-hungry. Prompt caching reuses that structure across requests, which is why re-sending the same long prefix can be fast and cheap while a fresh long context is neither. These three facts return in rung 4's vocabulary, where they start affecting real decisions.
You now hold the one job description that never changes across all five rungs: decide what deserves to be on the desk. At rung 1 you do it by hand, paste by paste. At rung 5 you do it by policy, deciding what an autonomous system may retrieve and retain. Same job. Different altitude.