Inside an LLM

The LLM itself: one file of numbers

Before the walk, the thing being walked through. An LLM ships as a weights file - typically model.safetensors, a format that is nothing more than a set of named arrays of numbers. Open GPT-2 small and the whole inventory fits on a screen:

That is the entire artifact, and it explains the shape of this article: every section below is a pass over that list in the order the numbers are used. Nothing in the file can run on its own - it needs a separate program to load the arrays and do arithmetic with them - and nothing in it changes while you use it. The same bytes answer your question and everyone else's, today and next year. Where the numbers came from is How a model learns; what happens to them in the next ten minutes is this walk.

None of this has to be taken on trust. GPT-2 is public, and the repository at huggingface.co/openai-community/gpt2 states every number used above: 124M parameters, stored as F32 - four bytes each, which is where the file size comes from - a vocabulary of 50,257 entries, and inputs of 1,024 consecutive tokens. Its Files tab is worth a minute on its own. model.safetensors sits beside config.json, which records the 768 columns and the 12 blocks, and beside vocab.json and merges.txt - the tokenizer's two files, shipped in the same repository but not inside the weights, and read by a different program before the model is reached at all. Where those two come from is What is BPE?

Scale changes the numbers in that inventory and almost nothing about its shape. A frontier model is the same list with roughly a hundred blocks instead of twelve, a few thousand columns instead of 768, a vocabulary two to four times larger, and the same total absence of code.

Width is worth a sentence of its own, because more columns is not a quality dial you can turn. The column count is d_model: how many numbers each token carries from the embedding table to the last block, and therefore how much one position can hold at once before distinctions start to overwrite each other. It is also what attention divides among its heads in section 06 - GPT-2 small splits 768 columns across 12 heads, 64 numbers each; GPT-3 splits 12,288 across 96 heads, 128 each. Wider is not a better answer so much as more room to carry one.

And width is charged by the square. A block holds roughly 12 times d_model squared numbers - the query, key, value and output projections, plus a feed-forward layer four times wider than the stream - so doubling the columns roughly quadruples the block, where adding a block only adds one block. Depth and width are raised together in rough proportion rather than either being pushed on its own, and what quality tracks is the total: parameters and training tokens, in balance. A model twice as wide trained on the same text is not twice as good, only four times as expensive per block.

The names in the left column are the only nouns that exist inside an LLM.

One question, one loop

What is an LLM? ends on one sentence: the model predicts the next token, and everything else is scaffolding. This article is what happens inside that sentence, for one question. Type What is the capital of the UK?, press Enter, and before the first letter of the answer appears the text has been wrapped in a template, cut into eight tokens, turned into eight lists of numbers, passed through dozens of layers in which every token reads every other, and scored against every entry in a vocabulary of over a hundred thousand. One entry wins. It is appended to the input, and the whole thing runs again.

That is the entire mechanism: a forward pass that produces one token, inside a loop that repeats it until the model emits a stop marker. Sections 03 to 09 walk through one pass. Section 10 is the loop. Section 11 is why a dozen things you have already noticed about these models fall out of the walk.

One rule for reading. The token splits shown are what a typical tokenizer produces; the IDs and probabilities are illustrative - they show the shape of what the model computes, not a measurement of any particular model. The shape is what to keep.

The prompt the model actually sees

You typed seven words and a question mark. The model receives more. Chat is a convention layered on a completion engine, and the convention is a chat template: the system prompt goes first, each turn is wrapped in role markers, and the text ends with the marker that opens the assistant's turn. One common shape, ChatML, looks like this:

Two details do most of the work. The markers such as <|im_start|> are special tokens - reserved entries in the vocabulary that the tokenizer never produces from user text, so typing one does not forge a turn. And the prompt ends mid-conversation: the last thing the model sees is an open assistant turn. The most likely continuation of a question followed by "assistant" is an answer. That is the whole trick that turns a text-completion engine into something that appears to converse.

Every vendor has its own markers, and the template is applied by the client library or the API, not by you - which is why you never see it. It still costs what any text costs: the system prompt and the markers are tokens, processed on every request before your question is. What's in an agent's context? covers what else ends up in this block once tools and documents join the conversation. For the rest of this article the prompt is the template above.

Tokenization: text to IDs

The network never sees text. A separate, small, deterministic program called the tokenizer converts the prompt into a list of integers, and that list is the only input the model gets. For the question itself the split is eight pieces:

Notice the leading spaces. " the", "the", and "The" are three different vocabulary entries, because the tokenizer was built by counting which byte sequences occur together most often in a large corpus and merging them, step by step, until the vocabulary reached its target size - roughly 100,000 to 200,000 entries for current models. That procedure is byte pair encoding. Common words become one token; rare words, code, and non-English text split into several; a word the corpus never saw still tokenizes, down to single bytes if it has to.

Three things follow that you will already have run into. Tokens are the unit of cost and of the context window, so the same sentence costs more in Polish or in JSON than in English prose. Counting the letters in a word is hard for the model because it never sees letters - " capital" arrives as one indivisible integer. And "UK", "U.K.", and "United Kingdom" are three different inputs, which the model has to have learned mean the same place.

Embeddings: IDs to vectors

An ID is an index, and the first layer of the network is a lookup: row 6560 of a table with one row per vocabulary entry. Each row is an embedding - a list of a few thousand numbers, learned during training. Nothing about the numbers is hand-designed. They ended up where they are because the training objective rewarded placing tokens that behave alike near each other, so " UK" sits close to " Britain" and " England", and " capital" sits close to " city" and, in another direction, to " money".

The table itself is an ordinary rectangle of numbers, and its two dimensions are decided by two different parties:

Two things are worth reading off that. The row count belongs to the tokenizer, which is why it has to be frozen before training starts - the rows are addressed by id, so changing which piece holds id 6560 would point every row at the wrong word. The vocabulary size a tokenizer settles on is paid for here, one row at a time. And the column count is not local to this layer: d_model is the width of every vector from here to the last block, so the table's columns and the residual stream in section 07 are the same number.

The same shape appears once more, at the far end. The scoring step in section 08 multiplies the final vector against a table with one row per vocabulary entry - this table's mirror image, read in the opposite direction. The embedding table turns one id into one vector; that one turns one vector into a score for every id. Some models train the two separately and some ship a single set of numbers used both ways, which is why a parameter count sometimes charges for this table twice and sometimes once.

This is the only lookup in the model. One row is fetched by id here, and everything between it and the answer is computed.- what the embedding table is, in one sentence

One more signal is mixed in before the vectors move on: position. The layer that follows treats its inputs as a set, not a sequence, so without an explicit position signal the capital of the UK and the UK of the capital would be indistinguishable. Most current models encode position by rotating the vectors by an angle that depends on where the token sits; the details vary, the purpose does not.

After this step the question is eight vectors of a few thousand numbers each, and every operation from here to the answer is arithmetic on them. The same idea - meaning as coordinates - is what a vector database stores and searches when it powers retrieval. Here it is the model's own internal representation.

Attention: "capital" finds "UK"

At this point each vector knows only its own token. Attention is the operation that lets it read the others, and it is the reason the architecture is called a transformer.

In plain terms, every position asks a question and every position offers an answer. From its vector, each token computes a query - what am I looking for - and a key and a value - what do I contain, and what will I hand over if asked. Each query is scored against every key; the scores are normalised into weights that sum to one; and the token's new vector is the weighted mix of everyone's values. A token that scores highly against a key pulls in a lot of that token's value. One that scores low pulls in almost nothing.

In the example, " capital" scores highly against " UK" and " of", so its vector leaves the layer carrying "capital, specifically of the UK". The position that matters most is the last one. The vector at the end of the prompt - the open assistant turn - is the one that will predict the next token, and it attends across the entire question. By the top of the network it no longer represents a role marker. It represents "the answer to a question about the UK's capital is about to be stated".

Each layer runs several of these attentions in parallel - heads - and each head is free to learn a different relation: which noun a pronoun refers to, which country a city belongs to, which bracket closes which. None of those roles are assigned. They emerge from training, and interpretability work spends much of its time working out which head learned what.

The cost is the tradeoff you pay on every request. Every token attends to every token, so the work grows with the square of the sequence length. That is why some vendors charge a higher rate above a context threshold, why a million-token window is an engineering achievement rather than a configuration value, and why section 10 needs a cache.

Layers: where UK to London lives

Attention is half of a transformer block. The other half is a feed-forward network: a small two-layer neural network applied to each position on its own, with no view of the neighbours. Attention moves information between tokens; the feed-forward layer transforms what each token now holds. Both write their result back onto the vector they read from, so the vector accumulates - the residual stream, in the interpretability vocabulary - rather than being replaced.

Then the block repeats. GPT-2 stacked 12 to 48 of them; models at the frontier stack on the order of a hundred. Early blocks resolve grammar and local meaning, middle blocks do most of the factual and relational work, late blocks shape the vector into something the output layer can score. The "70B" in a model's name counts the numbers inside these blocks: the embedding table, every head's query, key and value projections, every feed-forward layer.

This is where the fact lives, in the only sense it lives anywhere. Interpretability research that traces how GPT-style models recall facts points to the feed-forward layers in the middle of the stack: when the vector for " UK" arrives there carrying "the country whose capital is being asked for", those layers add "London" to it. The association is not stored as a row in a table. It is spread across weights that also encode every other capital, everything else written about the UK, and a great deal that has nothing to do with either.

There is no table of capitals. The answer is computed, not looked up.- the sentence to carry out of this section

Hold on to that when the same machinery meets a question it has thin evidence for. The layers that add "London" to "UK" will add something to any country, because adding a plausible continuation is what they do. Whether the something is right depends on how much of the training text said so - which is the subject of the next section.

Logits to probabilities

After the last block, only one vector is used: the one at the final position. It is multiplied against a second table with one row per vocabulary entry, which produces one score per entry - over a hundred thousand numbers called logits. Higher means "more likely to come next". A softmax turns the scores into probabilities: exponentiate each, divide by the total, and the result is a distribution over the whole vocabulary that sums to one.

For this question the distribution is extremely peaked. "The capital of the UK is London" appears in an enormous number of documents in every register - textbooks, travel pages, quiz sites, news - so the continuation is overwhelmingly determined. Compare a fact that appears in far fewer documents:

Both columns are illustrative - the shape, not a measurement. The shape is the point. On the left the top token wins by so much that nothing downstream can change the answer. On the right the runners-up are not noise: " Tar" begins Tarawa, the atoll the capital sits on, and " Ba" begins Bairiki, the islet older references name. Wrong-but-plausible alternatives with real probability mass are exactly what the mechanism produces when the training text was thin or inconsistent, and a sampler that picks one of them will do so with the same fluency as the correct answer. That is hallucination, seen from inside: not a mode the model enters, but a flat distribution followed by a pick.

None of this is hidden. Several APIs return the top few probabilities per generated token on request, and reading them for your own prompts is the fastest way to build intuition for where a model is sure and where it is guessing.

Sampling: temperature and why answers vary

A distribution is not an answer. One token has to be chosen, and the choice is the only place randomness enters the whole process. Three settings you have seen in every API control it.

• Greedy decoding takes the top token every time. Deterministic in principle, and prone to repetition on long outputs, because the locally most likely token is not always the globally best sentence.
• Temperature divides the logits before the softmax. Below 1 the distribution sharpens and the top token wins more often; above 1 it flattens and the tail gets its turn; at 0 the API falls back to greedy. It adds no knowledge - it only redistributes probability the model already assigned.
• Top-p (nucleus sampling) drops the tail first: keep the smallest set of tokens whose probabilities add up to p, then sample within it. Top-k keeps the k most likely instead. Either stops a one-in-ten-thousand token from ever being picked while leaving the genuine alternatives in play.

Look at the table again. On the left, no reasonable setting changes the first token - " London" at 0.97 survives any temperature you would actually use. What varies is the wording that follows: a low temperature produces "London." and a higher one produces "The capital of the UK is London." on some runs. On the right the setting decides the answer itself. At temperature 1 the runner-up comes out roughly one time in four; near 0 it almost never does; above 1 it does more often. Temperature is less a creativity knob than a "how often do I take the runner-up" knob, and the runner-up is only sometimes a synonym.

The practical rule follows. Extraction, classification, code, anything with a checkable answer: temperature low, or 0. Brainstorming, naming, variations on a draft: higher, with top-p to keep the tail out. And one caveat the vendors state themselves: temperature 0 makes output mostly repeatable, not guaranteed identical - batching and floating-point arithmetic on the accelerator introduce small differences that can flip a near-tie.

Append and repeat, until the stop token

" London" is appended to the list of IDs. Now there are nine tokens after the template instead of eight, and the whole pass - embeddings, every block, logits, softmax, sample - runs again to produce the tenth. Then again for the eleventh. Each pass yields exactly one token. "London." is two passes. A 300-word answer is around four hundred. When a chat interface streams the reply word by word, you are watching the loop iterate.

Three things about the loop are worth having straight.

It stops when the model says so, or when you do. The template in section 03 taught the model that turns end with <|im_end|>, so once the answer is complete that special token becomes the most likely next token and the loop exits. The other exits are yours: max_tokens, or a stop sequence you supplied. An answer cut off mid-sentence hit max_tokens. The model did not decide to stop; the loop was stopped.

Its only state is the list. Nothing inside the model changes between passes or between requests. The conversation exists as the list of IDs and nowhere else, which is the mechanical reason the 101 article could say the model cannot remember yesterday - and why a follow-up question works only because the client re-sends the whole exchange.

It does not redo all the work. Run literally, pass ten would recompute attention for all nine earlier tokens. Since their keys and values (section 06) do not change, implementations keep them in a KV cache and compute only the new position on each pass. That split is visible in two numbers every provider quotes: time to first token, which covers processing the entire prompt at once, and tokens per second, which is the cached loop running afterwards. A long prompt makes the first slow and leaves the second alone.

What this explains

Walk the pass once and a list of behaviours that look like quirks turn out to be consequences.

• Billing is per token because work is per token. The prompt is processed once, every output token is one more pass, and a long prompt is re-read on every turn of an agent loop. What an agent actually costs follows the money from here.
• Hallucination is not a separate mode. Section 08's flat distribution, section 09's pick, and section 10's loop continuing from whatever came out. The fix that works - putting the source text in the prompt - works because attention can read what is in front of it far more reliably than the feed-forward layers can recall it.
• Letters and digits are hard. The model never sees them (section 04). Spelling a word backwards or adding two long numbers means operating on tokens whose character content it has to have memorised.
• Wording changes the answer. Different tokens produce different vectors, different attention patterns, and a different distribution at the end. Prompting is input engineering, not persuasion - there is no one to persuade.
• Reasoning models are the same loop with more passes. They emit their working as tokens before the visible answer, so the final distribution is conditioned on text they wrote themselves. Nothing in the architecture changed; the loop got longer, and the bill with it.
• Tool use is text too. A tool call is a run of tokens the harness recognises and executes; the result is appended to the list as more text and the loop resumes. Functions, MCP, and Skills is the history of standardising that run of tokens.

If the 101 article's sentence was "it predicts the next token", this article's is one clause longer: it predicts the next token from a distribution that is computed, not retrieved, over everything it was shown. Every product decision in this Path is about what to show it.