Why AI Runs on GPUs

Zoom into one layer and follow one token through it. Most of the work happens in a few large steps.

Every one of those large steps is the same operation: multiplying two grids of numbers, a matrix multiplication.

Matrix multiplication is 99.8% of the arithmetic.

Add up all 32 layers and the final step that scores every word in the vocabulary, and one token costs about 17 billion operations.

Training multiplies this twice over: by the number of tokens the model reads, and by three, because learning also has to run the model backwards.

The question is not whether a computer can do this. It is how many matrix multiplications it can do per second.

A CPU has a handful of cores. Each is very fast and very clever, but it still fills the grid a few cells at a time.

A GPU has thousands of simple lanes. Each one is slower than a CPU core, but they all start at once, and the grid is done before the CPU is an eighth of the way through.

A CPU minimizes latency. A GPU maximizes throughput.

A GPU flips the ratio. The chip is tiled with copies of the same block, and most of every block is arithmetic.

Inside one block are four groups of 32 lanes, a small scratch space for each, and a special unit that multiplies whole small matrices at once.

The 32 lanes of a group do not act independently. They receive one instruction and execute it together, each on its own data.

So GPUs keep several layers of memory, each closer to the lanes and smaller than the last.

The closer the memory, the faster it is. Registers and shared memory move data ten to a hundred times faster than HBM, and answer in a few cycles instead of hundreds.

The whole game is to fetch each number from far away once, and use it many times up close.

Count how many operations a computation does for each byte it reads or writes. Few, and it waits for memory. Many, and it is limited only by the arithmetic.

A ReLU reads a number, compares it with zero and writes it back. One operation for every four bytes moved.

Matrix multiplication is different, because every number is used many times. Each entry of $A$ meets a whole column of $B$.

Try it. Small products fall on the slope, large ones climb onto the roof.

A GPU is only fast when each byte is reused many times.

Operations that cannot reuse data can still be combined. Do three elementwise steps in one pass, while the numbers are close by, and the intensity triples.

Computed naively, every output cell fetches its own row and column from main memory. Neighboring cells fetch the same numbers again and again.

Instead, a group of threads copies one block of $A$ and one block of $B$ into fast memory on the chip. Every number in those blocks now serves 4 outputs instead of one.

Then it slides along: the next block of $A$ to the right, the next block of $B$ below, adding to the same outputs each time.

Do this for every output tile and the trips to main memory drop from 1,024 to 256: 4 times fewer. Bigger tiles save more.

Tiling turns a memory-bound computation into a compute-bound one.

At the very bottom, the product of two small tiles is exactly what the GPU's matrix units were built for.

A computer stores a decimal number like scientific notation: a sign, a power that sets the scale, and a few digits of detail. The classic format uses 32 bits for all three.

Neural networks do not need seven digits. BF16 keeps all of the scale and drops most of the detail. π becomes 3.140625.

FP16 splits its 16 bits differently: more detail, less scale. Pick 70,000 and it overflows, because nothing above 65,504 fits.

Keep going. FP8 has 8 bits and FP4 only 4. π becomes 3.25, then 3, and 0.0001 vanishes to zero.

Every halving of the bits roughly doubles how many multiplications a tensor core can do per second, and halves how many bytes have to move.

Lower precision speeds up both ceilings of the roofline at once.

Reading your prompt is easy for a GPU: every word goes through the model at once, which makes big, efficient matrix products. Writing the answer is harder, because it happens one token at a time.

Reading the weights takes 4.8 milliseconds. The arithmetic takes a sliver of that. One user gets about 209 tokens per second, and the math units sit idle 99.7% of the time.

The fix is to serve many people at once. The weights are read once per step no matter how many sequences use them, so a batch of 32 gets nearly 32 times the throughput for free.

Batching is how one GPU serves thousands of users.

There is a catch. To avoid redoing work, each conversation keeps the keys and values of every token so far. This KV cache grows with every token, and it lives in the same memory as the weights.