What's inside an AI

A list and a recipe

The point: everything a language model does comes from running those numbers through those sums.

A model is the thing an AI company builds and runs. A language model works on text. Its list of numbers is called the weights. Picture each weight as a knob. Nobody sets them by hand. Training does, and that's the next chapter. The recipe says which numbers to multiply and add.

Guess the next token

The point: the sums end in a chance for every token of coming next. That guess is the model's one job. Repeat it and you get text.

A token is a word or a piece of a word. Think of your phone keyboard suggesting the next word. Press Take the top guess and that token goes on the end. Then the whole text runs through the sums again. Tap another word to swap it in.

From the moon, the top guesses say "a mixture of hydrogen and helium", which is false, and then repeat. No step in the recipe looks up or checks a fact.

A chatbot runs this loop with a far bigger model, one token at a time. Extra training makes it read like a helpful reply, and most add chance, so the top guess doesn't always win. assessed

Every token has an address

The point: sums need numbers, so each token gets an address, a list of numbers. Similar words end up close together.

Think of a theatre ticket: row 5, seat 12. Two numbers pick out one seat. A token's address uses hundreds of numbers. They are part of the weights, so training sets them too. Nobody told GPT-2 that Monday is near Tuesday, yet out of every token, Monday's four closest are other weekdays. measured

Dots and lines

The point: the recipe is knobs and valves. A valve lets one word change what another word does.

A neural network is the recipe drawn as plumbing. Each line is a pipe with a knob, its weight, that multiplies the number passing through. Each dot adds what arrives. A middle dot is also a valve, called a gate. At zero or less, it shuts and passes on 0. A knob marked −1 turns the number it multiplies negative.

"Not good" means bad, but "not bad" means good. So "not" can't simply push the answer one way. It has to flip whatever comes next. A knob alone can't do that: it gives the same push every time.

This tiny network guesses whether "The food was ..." is pleased or unhappy. It has two inputs, good and not. Each is 1 if its word is there and 0 if it isn't. To keep it small, bad is just good switched off. The top gate opens for good without not. The bottom gate opens for not without good.

Press Add "not" on good, then on bad. "Not" pulls the answer down the first time and up the second. Then press Take out the gates. With nothing to shut, the two middle sums cancel and the answer is 0 for every sentence. In this toy, the gates are what let a word's effect depend on the words beside it.

GPT-2 is the same recipe, much bigger. Addresses go in. Dots side by side form a layer, and there are 12 layers in a row. After the last one, the sums give every token a score, and the scores become the chances you saw. assessed

Same word, different numbers

The point: the stored address is only where a word starts. On the way through the layers, its numbers shift with the words around it.

In the toy, each word was a fixed input. GPT-2's layers do one more thing. Each layer lets a word's numbers borrow a little from the words before it in the sentence, so a word can change depending on what came earlier.

Every "bank" starts from the same stored address. In "The angler walked down to the bank", bank picks up numbers from the words before it, and so it does in "The teller walked down to the bank". The two banks end up with different numbers.

I wrote 40 sentences like that, in pairs that differ by one word, and followed each "bank" through GPT-2's 12 layers. Press Play and watch which side each dot ends up on.

Keep this one idea: the same word gets different numbers in different sentences, mostly because of the words around it. That shows the numbers follow context. It doesn't show the machine knows what a river is.

Water through pipes

The point: reading your sentence doesn't change the weights, so nothing you type gets filed away in them.

Your sentence is water. It flows through, makes a guess and drains away. The knobs stay put.

So is your chat private? Not for that reason. A company may keep your chat and train a later version on it, which can fold text into the weights. Anthropic, which made me, said in 2025 that consumer users can choose. third-party

"Just sums" explains less than it sounds

The point: knowing the parts doesn't tell you what the whole can do. A brain is cells firing, and that settles little.

"When Kate and Mark went to the market, Mark gave a hammer to ___." You said Kate. Swap the names and you'd say Mark. The last few words match both times, so copying phrases can't get both right. GPT-2 rated the right name higher in 99.2% of 600 such sentences. measured That doesn't show reasoning. One simple rule also passes: answer the name that appears once.

What you can fairly say