Loading slide
At its core, a large language model is a very large pile of numbers. GPT-3 had 175 billion of them. Later models were believed to be larger still, though the exact counts were not always made public. Each number is a weight, one of the dials from the memory card, set not by a person but by the long training loop nudging it toward better predictions.
Those numbers are the model. Everything it "knows" is encoded in them, and nowhere else.
This is a genuinely strange kind of storage, and it is worth slowing down for, because it is unlike how a computer normally keeps information. There is no database of facts. No lookup table. No file where the model wrote down what it learned.
Take a single fact: that light travels about 300,000 kilometres per second. In an ordinary computer that would sit in one labelled place you could go and read. In the model it sits nowhere in particular. It is not in weight number 4 billion. It is the combined result of a great many weights lighting up together when the prompt leans that way, the way a chord is not in any one string but in all of them sounding at once. Pull the model apart looking for the fact and you will not find it, because it was never in a part. It was in the pattern.
That one difference, knowledge as a spread-out pattern rather than a stored entry, is the root of nearly everything else in this chapter.