What the Weights Contain

A language model's knowledge is not kept the way you would probably imagine.

Not in a list of facts. Not in a database you could search. Not in anything you could open up and read line by line. It is spread across billions of numbers, the weights, each one nudged a trillion tiny times during training until the whole arrangement, taken together, got good at predicting text.

Those numbers are everything the model "knows." But knowing, for a model, is a strange business. A fact does not sit in one place. It lives as a pattern smeared across the whole network. The model can recall it, fail to recall it, get it right in one conversation and wrong in the next, and have no awareness of any of that happening.

That odd kind of knowledge is the source of both the impressive capability and the baffling failures, often from the same model in the same breath. To see why, we have to hold a different picture of what it means for a machine to know something. This chapter builds that picture, and by the end the hallucination we met earlier will look less like a bug and more like the shadow cast by the model's greatest strength.