A CPU has a handful of very fast cores. Give it a complicated job with lots of if this, then that and it is superb. A GPU has thousands of much simpler ones. Give it the same small sum to do a million times over and it will finish while the CPU is still rolling its sleeves up.
That is the entire difference. Not better or worse — a different shape, for a different kind of work.
Underneath the conversation, a language model is doing one thing over and over: multiplying long lists of numbers together and adding up the results. Billions of times, for every answer. No branching, no cleverness — the same small sum, endlessly.
Which is the job a GPU was already built for. Drawing a frame of a game means working out a colour for two million pixels at once. Running a model means working out a number for billions of weights at once. The chip could not care less which one you asked for.
A large model does not fit on one card. The weights alone run to hundreds of gigabytes, so the model is split across several GPUs that have to talk to each other constantly — which is why the cable between them matters as much as the chips.
And the expensive part is not the sum, it is fetching the numbers to do it with. Most of the effort in making models fast goes into moving less data, not into doing less arithmetic.
