What the Weights Contain

A language model's knowledge is not stored the way you might imagine.

Not in a list of facts. Not in a database you could query. Not in anything that could be opened up and read. It is distributed across billions of numbers, each one adjusted trillions of times during training until the whole arrangement became good at predicting text.

Those numbers are everything the model "knows." But knowing, for a model, is a strange thing. A fact is not stored anywhere in particular. It lives in a pattern spread across the network. The model can recall it, fail to recall it, recall it correctly in one context and incorrectly in another, and have no awareness of any of those things.

That kind of knowledge produces both impressive capability and baffling failure, often from the same system in the same conversation. Understanding why requires holding a different picture of what it means for a machine to know something.