What a Trained Model Actually Is

After training, something remains. But it is not what most people picture.

Not a list of rules. Not a database of facts. Not a set of instructions someone wrote down. What remains is a very large collection of numbers, arranged in a configuration that the training process arrived at through billions of small adjustments.

Those numbers encode, in a form no human can directly read, whatever patterns the model extracted from its training data. The knowledge is real. It is also strange. It cannot be opened up and inspected the way a filing cabinet can. It does not live in any one place. It is distributed across the whole.

This is the thing about learned systems: they work, often impressively, but the reason they work is buried inside something that resists explanation. That gap between capability and legibility is not a temporary problem waiting to be solved. It is, in some deep sense, what learning produces.