The one thing it does
A large language model does exactly one thing: given a stretch of text, it predicts what comes next. Not "understands the question and retrieves the answer" - predicts the next fragment of text, then does it again with that fragment included, and again, until it produces a stopping signal.
That is the whole mechanism. Everything that feels like conversation, reasoning, or expertise is what that one operation looks like when the model performing it has been trained on an enormous amount of text and has billions of internal parameters to work with.
It is worth sitting with how strange this is. Nothing in the system is looking anything up. There is no database of facts, no store of the documents it learned from. There is a very large function that takes text and returns a probability for every possible next token - and the fact that this produces working code, decent translation and a plausible explanation of photosynthesis is the genuinely surprising result of the last few years, not an implementation detail.
It is a text-completion engine. Every product built on one is scaffolding around that single operation.- the sentence the rest of this path depends on
Tokens, not words
The model does not see letters or words. Before anything happens, text is split into tokens - chunks that are usually a common word, a word fragment, or a piece of punctuation. "The capital of France is" becomes five tokens; an unusual name might take three on its own.
A rough working figure for English is that a token averages about four characters, so 1,000 tokens is somewhere near 750 words. Code, other languages, and unusual formatting all shift that ratio, sometimes a lot.
Tokens are not a technicality you can skip past, because three things you will meet constantly are denominated in them:
- Price. Providers bill per token in and per token out, which is why what an agent costs is a question about tokens rather than about requests.
- Limits. Context windows and output caps are token counts, not word or character counts.
- Odd behaviour. Some classic failures - miscounting letters in a word, mangling arithmetic on long numbers - are much easier to understand once you know the model never saw the individual characters.
Training and inference
Two entirely different activities get called "the model doing something", and confusing them causes most beginner misconceptions.
Pretraining
The model reads an enormous body of text and adjusts billions of parameters to get better at predicting the next token. Months of compute. The result is a frozen file of numbers.
Post-training
Further training shapes the raw predictor into something useful and safe to talk to - following instructions, refusing certain requests, adopting a consistent manner.
Inference
Your actual request. The frozen parameters are read, never written. Nothing you type changes the model, and nothing it says is saved into it.
The consequence people miss: the model does not learn from you. A correction you make in a conversation influences the rest of that conversation and nothing beyond it. It also means the model's knowledge stops at whatever date its training data ended - which is why a model with no web access will confidently discuss a world that is months out of date.
The context window
Since inference changes nothing permanently, everything the model knows about your situation has to be present in the text you send it. That text is the context window, and it has a hard size limit measured in tokens.
Everything competes for the same space: the system prompt that sets behaviour, the conversation so far, any documents pasted in, any tool definitions, and the reply being generated. A "1 million token context window" sounds unlimited and is not - a long agent run fills a large window surprisingly fast, which is the mechanism behind the context exhaustion described in Why agents fail.
Two practical effects worth carrying around. Position matters: material at the very start and very end of a long context tends to be used more reliably than material buried in the middle. And cost scales with what you send, every time you send it - a long conversation is re-transmitted in full on every turn, because the model has no other way to know what was said.
Why it invents things
A model producing a confident, well-formed, entirely false answer is not malfunctioning. It is doing precisely what it was built to do, applied to a case where the truthful answer was not available to it.
The mechanism follows from section 01. Asked for the title of a paper by a researcher it knows little about, the model has no representation of "I do not have this" to fall back on - it has a distribution over plausible next tokens. Academic titles have a recognisable shape, so it produces one with the right shape. The output is not retrieved and then corrupted; it is generated the same way every correct answer is generated.
Which explains the pattern users find most unsettling: fluency is unrelated to accuracy. A fabricated citation reads exactly as well as a real one, because both are produced by the same process, and nothing in the mechanism makes the model less certain-sounding when it is on thin ice.
Two things reduce it, neither completely. Giving the model the source material in context - search results, your documents, tool output - moves it from recall to reading, which it is far better at. And asking for verifiable outputs, so a claim can be checked rather than trusted.
Why it cannot remember yesterday
Each request is answered from nothing but the text in that request. There is no session, no accumulated impression of you, no yesterday. A model is stateless in the strictest sense.
Chat products feel otherwise because they re-send the conversation every turn. Ask a follow-up question and the interface quietly transmits the whole exchange again, so the model can behave as though it remembers. When a long conversation starts losing the thread, this is usually why: the earliest turns have been dropped or summarised to fit the window.
Product-level "memory" features work the same way, one layer up - notes are stored outside the model and inserted into the context when relevant. Useful, and worth understanding for what it is: retrieval and re-sending, not the model changing.
Everything else is scaffolding
With the engine understood, the rest of the field arranges itself neatly. Almost every AI product is a way of choosing what text to put in front of a text-completion engine, and what to do with the text that comes out.
- Chat interfaces re-send the conversation each turn so a stateless model appears to hold a thread.
- Retrieval searches your documents and pastes the relevant parts into the context, so the model reads rather than recalls.
- Tool use lets the model emit a structured request that your code executes, feeding the result back as more context.
- Agents put that tool loop under the model's own control, letting it decide the next step until a goal is met.
None of those change the engine. They change what it sees and what happens to what it says - which turns out to be enough. If you keep one idea from this article, keep the one from section 01: it predicts the next token, and everything else is arrangement around that.