Two families, and the analogy that fits neither

Almost everyone meets image generation through the same analogy: a language model predicts the next word, so an image model must predict the next pixel. The first half is a fair summary of Inside an LLM. The second half is wrong, and no single sentence replaces it, because there is no single way to make a picture. There are two, and they work nothing alike.

  • Diffusion is the older family and still the wider one: Stable Diffusion, Midjourney, the DALL-E line. It starts from a canvas of random static and cleans it up, pass after pass, until a picture is left. The whole canvas is worked on at once, and nothing is settled until the last pass.
  • Autoregressive is the newer family, and the one most people are using without being told: GPT Image in ChatGPT, very likely Nano Banana in Gemini. One model reads your text and your reference images and emits the picture itself, token by token, in the same stack it thinks in - a token being a patch of picture, which is where the analogy breaks: the sequence is real, the pixel is not.
Diffusion Autoregressive
How it builds The whole canvas at once, cleaned up over N passes Token by token, in sequence, exactly like text
The prompt Handled by a separate network, consulted each pass Read by the same model that emits the image
Editing Masks, strength settings, control networks A follow-up turn in the conversation
In-image text Historically the weakest point Advertised as a headline capability
What you pay for Steps x resolution Tokens, by size and quality tier

One note on vocabulary. The word both vendors reach for is native - OpenAI launched the generator built into ChatGPT as a "natively multimodal model", Google calls Nano Banana native image generation in Gemini - and neither phrase names an architecture. OpenAI states the architecture separately, and states it as a break: unlike the diffusion models that preceded them, DALL-E 2 and DALL-E 3, the GPT Image models are autoregressive - which is why the DALL-E line sits in the first bullet above and its successor in the second. Google has never said what Nano Banana's is, so the family above is a reasonable inference rather than a published fact.

Two families, two ways of building a picture. Neither of them paints one pixel at a time.- the one sentence to keep

What diffusion actually is

In plain terms: a diffusion model learns to reverse noise. Training runs one direction - take a real image, one of billions in the training set, and add Gaussian noise step by step until nothing of it is left. Generation runs the other - start from pure noise and remove a little at a time until a picture appears. The prompt rides along as conditioning, steering every step toward what you asked for.

The name comes from physics. Drop ink into water and it spreads until the glass is uniformly grey: information destroyed in a way that can be described exactly. Doing that to an image takes nothing but arithmetic, and no model is involved. Undoing it is the half nobody can write down, and it is the only reason a network exists here at all.

Which raises the obvious question: if the model only ever removes noise, why does anyone add it? Because the two directions run at different times, and only one of them runs for you.

  • In training, both directions run. Adding the noise is how the practice material gets made: damage a real image by a known amount, ask the network to name the damage, correct it by how wrong it was. The known amount is the answer key - that is the only reason noise gets added at all.
  • When you generate, only the removing half runs. There is no picture to damage. The run starts from pure noise and the model subtracts its way toward an image, pass after pass, with your prompt steering each one.

So a diffusion model is really a model of the reverse of a diffusion process. The forward direction is the scaffolding it was taught with, not something it does to your prompt.

Which also settles the question people ask next: generation is not the model reopening a training image. Once training ends those images are gone, and what is left is a few gigabytes of weights standing against billions of training examples - well under a byte each, with nowhere for the pictures to be. What it learned is what an undamaged image looks like in general, and a fresh field of static resolves into one that has almost certainly never existed. Duplicated images are the exception worth knowing about: something that appeared many times over in the data can come back close to verbatim, which is what the extraction work on diffusion models demonstrates.

One piece of vocabulary, because the two terms get used as synonyms and they are not. Diffusion is a technique, not a category. An image model is anything that produces images; diffusion is the most common way to build one, not the only one. The autoregressive family from section 03 is the live alternative; generative adversarial networks held the field before diffusion, and the pixel-at-a-time models of 2016 came earlier still. Every diffusion model is an image model; not every image model is a diffusion model.

Adding the noise is arithmetic. Removing it is the model. Diffusion is the name of the round trip.- the word, defined
A dark walnut desk under a brass lamp. A long brass rail holds seven framed glass plates in a row, each showing the same photograph of an orange tabby cat sitting on a sunlit wooden floor in front of a tall window. The leftmost plate is sharp; each plate to the right is buried in more grey static until the rightmost is pure noise with no trace of the cat left. Engraved brass number plates beneath the frames read 0, 5, 10, 15, 20, 25 and 30. A brass arrow above the rail points left to right, engraved ADD NOISE - JUST ARITHMETIC. A second arrow below the rail points right to left and glows orange along its length, engraved REMOVE NOISE - THE MODEL. A larger nameplate at the front edge of the desk reads Diffusion.
One scene, seven noise levels · left to right needs no model, right to left is the only reason one exists

Which leaves the question everyone actually arrives with: how does a prompt become a picture? Here is one request, every stage in order, with what exists on the machine after each one.

Text "orange cat in the house" · end to end
1 text encoder the sentence -> some vectors no image yet 2 seed a small grid of random numbers pure static 3 step 1 of 30 denoiser reads ALL of it, twice static, less (once with the prompt, once without) 4 steps 2..30 same field, same vectors shapes, then nothing is committed at any step detail 5 decoder the small grid -> 1024 x 1024 the pixels 6 encode PNG bytes out ~60 network passes and one decode. Not one of them saw a pixel.
A dark walnut desk under a brass lamp, six brass instruments in a row like stations on a workbench, each with an engraved number plate in front of it, bookshelves and ivy out of focus behind. Station 1: a handwritten card reading orange cat in the house beside a brass chamber engraved TEXT ENCODER, copper wires leaving its side. Station 2: a glass jar filled with pure grey static. Station 3: a tall glass tank engraved DENOISER holding dense static with the faint shape of a sitting cat barely visible in it, two copper pipes entering the top labelled WITH PROMPT and WITHOUT. Station 4: an identical tank engraved x30 in which the picture is fully resolved, an orange tabby cat sitting on a sunlit wooden floor in front of a tall window, a thick copper pipe looping from the top back into the base and glowing orange. Station 5: a brass press engraved DECODER. Station 6: a finished photographic print of the same cat lying flat on the desk. A long nameplate along the front edge of the desk reads One prompt, thirty passes, one print.
One prompt, six stations · stations 2 to 4 are the same small field, redrawn thirty times

Two things in that walk surprise people. The prompt is consulted every step rather than once at the start, because the denoiser has no memory between steps - the vectors from stage 1 are handed to it again on every pass. And the order in which the picture arrives is fixed by how much noise is scheduled for each step, not by the subject: the early steps settle composition and large shapes while the field is still mostly static, and the late steps settle texture and edges. That is why a render that is going wrong is usually wrong by step five, and why stopping early gives you a blurry version of the same composition rather than a half-finished corner of a good one.

What autoregressive actually is

Now the other family, and the one most readers are actually using. An autoregressive image model writes a picture the way a language model writes a sentence. Image tokens sit in its vocabulary alongside words. It reads your prompt, emits one image token, then the next, each conditioned on everything before it, until the picture is done. A decoder turns the finished sequence into pixels. No noise, no canvas, no steps.

The unit is a patch, and that is the detail the pixel analogy misses. One pixel at a time would mean over a million forward passes for a single 1024x1024 image, which nobody can afford. Make the unit a patch and the same picture is a few thousand tokens - a sequence length a transformer handles every day. It is the same bargain BPE strikes on the text side: pick a unit coarse enough to be affordable and fine enough to be expressive, then learn a fixed vocabulary of them.

Which means everything from Inside an LLM applies here unchanged: attention over the whole sequence so far, sampling with a temperature, a context window holding the conversation. And it all follows from one fact - your prompt and the picture live in the same sequence.

  • Instructions land. "the third cat, not the first", "put it by the window, smaller" - relative placement and counting are exactly the kind of constraint attention is built to carry.
  • Editing is a follow-up turn. A reference image is just more tokens in the context, so "same cat, different room" is a message in a conversation rather than a mask and a strength setting.
  • In-image text is far better. The string you asked for is present as tokens in the model that draws, instead of being a texture reconstructed from scratch.
A language model with pictures in its vocabulary. That is the whole idea.- the second family, defined
Text "orange cat in the house" · the other family
1 one model reads the prompt, and any reference images 2 token 1 one patch of picture, from a learned vocabulary 3 token 2..N each conditioned on every token before it a few thousand of them for a 1024 x 1024 render 4 decoder the finished token grid -> pixels, one pass 5 encode PNG bytes out One stack, one pass down the sequence. Nothing was denoised.

The costs are diffusion's in reverse. The picture arrives one token after another, so it cannot be worked on all at once the way a canvas can, and the bill is a token count rather than a step count (section 08). There is no open checkpoint to download either, so none of the diffusion toolbox exists here: no control networks to constrain the layout, no small style add-ons to load on top, no seed to pin for an exactly reproducible render. You get instruction-following instead of control.

One honest boundary, because this family is documented far less than the other. That GPT Image is autoregressive is stated: OpenAI's own system-card addendum describes its native image generation that way, and it is the clean break with the DALL-E line. Everything under that is not published - how the image vocabulary is built, how many tokens a render really uses, what the final decode is made of. For Nano Banana even the family is inference: Google calls it native image generation in Gemini and names no architecture. So take the walk above as the shape of the thing, not a schematic.

What is actually in the file

Worth knowing before the next two sections, because they name these parts: "the model" is not one network. An open diffusion model ships as a single .safetensors checkpoint - which is why the community trades them as single files - and inside that file sit several separate networks with separate jobs, trained at different times.

Text one open checkpoint · Stable Diffusion 1.5, the public reference
text encoder ~123M params your sentence -> vectors once or twice denoiser ~860M params the U-Net, the big one runs N times autoencoder ~50M params the decoder half runs once scheduler 0 params a table of noise levels just a list roughly 1B parameters in total, a few GB on disk # Frontier models are larger in every row. The inventory is the same shape.

Only one row is what people mean by "the image model": the denoiser. It is the biggest, and it is the only one the loop runs more than once - fifty times in the published mobile benchmark of this checkpoint, against a couple of runs of the text encoder - the second for the prompt-free branch - and a single run of the decoder. The scheduler carries no weights at all; it is a list of how much noise belongs at each step.

Two of those rows are swappable, which is why the open ecosystem works the way it does: the same autoencoder or text encoder can be paired with a different denoiser. If you ever read the code, you will meet them by their class names - CLIPTextModel, UNet2DConditionModel, AutoencoderKL, plus a scheduler - because a pipeline is assembled from exactly these pieces.

Through a hosted API none of this is visible, and for the autoregressive family it is not published at all. GPT Image and Nano Banana are endpoints: no file, no parameter counts, and no statement of what the image vocabulary or the final decode is made of. Everything in this section is the diffusion side, because the diffusion side is the only one you can open.

Several networks ship as one file. Only one of them is the thing people mean by "the model".- what a checkpoint really is

Training: what each family is scored on

Training is where the two families diverge furthest. Both are handed a real image, both read the prompt on the way, and both end at a score - but what happens to that image is different, and so is what the score measures. One is asked which token comes next; the other is asked how much noise it is looking at.

Autoregressive Diffusion PIECE BY PIECE · THE PUBLISHED SHAPE NOISE TO PICTURE A real image one of billions Image encoder picture -> image tokens Decoder-only transformer reads the prompt tokens too Predict the next token scored at every position at once loss: the tokens it should have emitted A real image one of billions Add noise a random amount of it Denoiser reads the encoded prompt too Predict the noise scored against what we added loss: the noise we actually added
Both start from a real image and both end at a number to be scored - but one is asked which token comes next at every position and the other which noise was added. The diffusion column is the objective from section 02 exactly; the autoregressive column is the shape the open literature describes, since neither vendor documents its own image vocabulary.
A dark walnut desk under a brass lamp. Two brass instruments stand side by side in lit glass cases, bookshelves and ivy out of focus behind. On the desk in front of both lies one photographic print of an orange tabby cat sitting on a sunlit wooden floor, on a plate engraved TRAINING EXAMPLE, with a handwritten card beside it reading orange cat in the house and a copper wire running from the print into each case. The left case, engraved AUTOREGRESSIVE, shows the same photograph cut into a row of small square tiles along a brass rail, the first tiles in place and a brass arm holding the next tile above its empty slot; below the rail a small brass balance holds a picture tile on each pan, the pans labelled PREDICTED and ACTUAL, with a plate beneath reading LOSS. The right case, engraved DIFFUSION, shows the same photograph inside a frame, half buried in dense grey static, with a large brass dial beside it engraved RANDOM AMOUNT; below it an identical balance holds a pane of grey static on each pan, labelled PREDICTED and ADDED, with a plate beneath reading LOSS. A long nameplate along the front edge of the desk reads Scored, not taught to draw.
One training example, two questions asked of it · each family is scored against a different answer key

Neither is taught to draw. Both are taught to answer a question about material they were given, over and over across billions of examples, and drawing is what falls out of being able to answer it.

A request: what each family repeats

Running a request asks the same question, now with no answer key. Nothing new is learned; the question the network was trained on is simply asked over and over until there is a picture. For an autoregressive model that is the identical operation in the identical direction - predict what comes next, from what exists so far. For diffusion it runs the process backwards: training added the noise, a request removes it. What differs after that is how much gets settled along the way, and the table in section 01 sets out the rest.

Autoregressive Diffusion PIECE BY PIECE · THE PUBLISHED SHAPE NOISE TO PICTURE Your prompt tokens in a sequence Decoder-only transformer emits token 1, 2, 3 ... N Decoder token grid -> pixels, once The final image nothing revisited one pass down the sequence Pure noise the seed, no content yet Denoiser, x30 the whole canvas, every pass Decoder compact grid -> pixels, once The final image settled at the last pass 20 to 50 passes, then one decode
The same objective, now run to produce something. One commits each piece as it goes; the other keeps everything provisional until the last pass. Both end at a decoder that touches pixels exactly once. As in section 05, the autoregressive column is the shape the open literature describes - the vendors document neither their token vocabulary nor what the final decode is made of.
A dark walnut desk under a brass lamp. Two brass instruments stand side by side, each in its own lit glass case with a slim brass divider between them, bookshelves and ivy out of focus behind. The left case is engraved AUTOREGRESSIVE and holds a horizontal brass rail of small square picture tiles: the tiles on the left are filled in and together show part of an orange tabby cat sitting on a sunlit wooden floor, while the tiles to the right are still empty dark squares, with a brass arm poised above the first empty one. A plate below it reads ONE PASS. The right case is engraved DIFFUSION and holds a single tall pane showing the whole picture at once, the same orange tabby half emerged from dense grey static, with a copper pipe looping from the top of the case back into its base, glowing orange and engraved x30. A plate below it reads EVERY PASS. In front of both cases a handwritten card in a brass holder reads orange cat in the house, with a copper wire running from it into each case. A long nameplate along the front edge of the desk reads Same request, two ways to answer it.
One request, two answers · the left case commits a piece at a time, the right keeps the whole picture provisional

If an analogy helps: autoregressive is drawing stroke by stroke in sequence, each stroke committed and built on. Diffusion is closer to a canvas full of static with a picture gradually revealed inside it - broad structure first, finer and finer detail after - with every part of it reworked on every pass. Nothing is sketched; something is uncovered.

Why the same prompt gives a different image

Because the prompt is not the input. The input is your prompt plus a field of random numbers, and the prompt only steers what happens to that field. Ask twice and you get two different pictures that both satisfy the same description - not a model being inconsistent, but a different starting point being resolved by the same rules.

That field has a name and an address. The seed is the integer that generates it, and it is literally what a run starts from. Pin the seed and the variation disappears: same seed, same prompt, same number of steps, same guidance setting - the dial for how hard the prompt is pushed - same model version, and the same picture comes back. That is as close to determinism as image generation gets, and it is the reason "seed" is a field in every serious diffusion UI.

Text the same image twice · what has to be held still
seed the starting noise pin it prompt the exact string, same encoder pin it steps same number of them pin it guidance same strength setting pin it model same weights, same version pin it size same output dimensions pin it change any one row -> a different picture, not a variation of one the multimodal APIs expose no seed -> send the image back as reference

So "the same prompt gave me something else" is usually two separate things wearing one complaint. One is the seed, drawn fresh on every run unless you pin it. The other is the prompt itself: most of a typical request is never actually specified at all, and every detail you leave out is one the prior gets to choose - a new seed simply lets it choose differently. A tighter prompt narrows the range of pictures a new seed can produce; it does not make two runs identical.

Two caveats keep this honest. Even a pinned seed is only reproducible within the same stack - a different GPU, a different implementation of the same steps, or an upgraded library can shift the result slightly, because the arithmetic is not bit-identical everywhere. And most hosted endpoints do not expose a seed at all, so exact reproduction is simply not on the menu.

The autoregressive family varies for a different reason, and section 03 already named it: the picture is sampled token by token from a distribution, exactly the way text is. Two runs diverge at the first token that could have gone either way, and nothing downstream pulls them back together. Neither GPT Image nor Nano Banana gives you a seed to hold, which is why consistency in that family is a workflow rather than a setting: generate once, then pass that image back in as a reference and edit it in follow-up turns, so later pictures inherit the first one instead of being redrawn from nothing. That is also where the vendors are spending their effort: OpenAI's ChatGPT Images 2.5 release leads on holding reference subjects recognisable and keeping edits stable across several turns, rather than on anything you could pin with a number.

A prompt describes a set of acceptable pictures. The seed picks which one you get.- the honest version of "be more specific"

What you are billed for

The one real difference from text billing is whether you can know the bill in advance. With text you cannot: the model decides how long the answer runs, so the output count only exists once it has finished. With an image you can. Pick a size and a quality tier and the number of output tokens is already fixed - the same count on every request, known before the model runs.

Text billing is two numbers: tokens in, tokens out. Image billing is the same two numbers. Nothing here is priced per picture.- what the meter actually counts

Which makes the unit from What is a token? the unit of the invoice here too, and four properties of that meter are worth having straight before you put a render in a product.

  • The price is settled before the first step. On the hosted APIs you choose a size and a quality tier and that fixes the cost; if you run a diffusion model yourself the cost is steps times resolution times passes per step, and guidance makes that last factor two, because it runs the network twice per step: once with your prompt, once without. Either way there is no equivalent of "the model decided to write four paragraphs".
  • Difficulty is still free. The token count is fixed by size and quality tier before the model runs, so a hard composition and a trivial one at the same settings bill identically - exactly as a hard question and an easy question of the same length cost the same in text. Only three things move the bill: the quality tier, which spans roughly 35x from low to high; the output size; and reference images on an edit. Nothing about the content of the request appears on the invoice.
  • Editing is not a discount on generating. Both families re-run the whole pipeline for an edit, and the input image is charged on top - as input tokens on a multimodal API, as an extra encode plus a full sampling run on your own diffusion stack. GPT Image 2 always processes reference images at high fidelity, so edit-heavy work runs two to three times the baseline per image. "Just move the cat to the window" costs more than the original render, not less.
  • Retries are the real budget line. Image work is iterative by nature, so the number that matters is cost per accepted image, not cost per call - the same correction What an agent actually costs makes for agent runs. Four rejected drafts at draft quality are usually cheaper than one at maximum quality, which is why every serious pipeline drafts low and renders high once.
The meter counts the size of what comes out, never the difficulty of what you asked for. Asking for something harder is free; asking for more of it is not.- the whole idea of image billing

The shape of it in real numbers, for the generation whose figures are published and checked: GPT Image 2 and the Nano Banana line, September 2026.

Per image How it is metered
GPT Image 2 · 1024x1024 $0.006 low · $0.053 medium · $0.211 high output tokens at $30 per 1M, plus $8 per 1M for input images
Nano Banana Pro about $0.13 at 1K to 2K · $0.24 at 4K 1,120 output tokens at 1K to 2K, 2,000 at 4K, at $120 per 1M
Nano Banana 2 about $0.045 to $0.15 output tokens at $60 per 1M, the count rising with resolution
Nano Banana 2 Lite about $0.034 at 1K 1,120 output tokens at $30 per 1M

Three mechanics that the rates alone do not show. Bigger is not reliably dearer: on GPT Image 2 the portrait and landscape sizes bill slightly below square at medium and high, because a larger non-square image can use fewer tokens - so cost cannot be extrapolated by multiplying pixels. Batch requests run at half rate with a 24-hour turnaround, for both generations and edits. And a failed generation is not billed, which makes the retry maths above about rejected drafts, not broken calls.

One caveat on the table itself: it is a generation behind the newest models. OpenAI's Images 2.5 release adds GPT-Image-2.5 Flare and Sunburst in the API, and its quality levels are not one-to-one equivalents of GPT Image 2's low, medium and high - so their rates have to come from OpenAI's own pricing page rather than being guessed at from the rows above. The mechanism does not change: a size and a tier fix a token count, and that count times the rate is the bill.

The fuller model-by-model table lives in the cost section of the GPT Image article. Read any of these for ratios rather than cents: prices move, and the durable fact is that a high-quality render costs tens of times a draft, and that gap is the whole of image cost engineering.

References