Can anyone look inside?

Everything so far has been a list of things you cannot do. That is not the whole picture, and the exception is worth knowing about.

People do try to read these machines. The field has a name, interpretability, and its aim is to work out what is happening inside a trained model rather than accepting the outputs and shrugging.

The obvious approach fails immediately. Pick one unit inside the model, watch when it fires, and see what it responds to. Researchers did this on an image model years ago and found a single unit that responded to cat faces and to the fronts of cars. Not a bug, and not a hidden connection between cats and cars. That unit was simply carrying both, along with plenty else.

Which is superposition again, seen from the outside. If concepts are sharing space, then looking at one unit shows you a mixture, and mixtures are not readable.

But sharing has a mathematical shape, and a mixture that was combined can in principle be separated. Recent work does exactly that: rather than asking what each unit means, it looks across many units at once and tries to recover the individual ingredients that were mixed together. Ingredients found this way are far cleaner. Instead of a unit that means cats and cars, you get something that responds to one thing, and one thing only.

In 2023 this was shown on a deliberately tiny model, one layer deep, which left open the question of whether it would survive contact with anything real. In 2024 the same team ran it on a production model and reported millions of these ingredients.

Some are what you would expect: a specific person, a country, a kind of code. Others are less obvious. One responds to the Golden Gate Bridge, whether the bridge is described in English, mentioned in another language, or simply shown in a picture, which tells you it is not matching a word. Others track things like a bug in a piece of code, or flattery, or a text being deceptive.

And there is a way to check these are real rather than a story told about numbers. You can turn one up. Force the Golden Gate ingredient to stay strongly active and the model will steer whatever it is talking about back toward the bridge, unprompted and continually. It moves the claim from this number seems to correspond to the bridge to this number causes the bridge, which is the difference between noticing a pattern and finding a part.

So the honest position is neither of the two you usually hear. It is not true that nobody has any idea what happens inside. It is also not true that we can read a model the way you read a program. Researchers can now find some of what is in there, with considerable effort, and account for part of what a model does. The rest is still dark, and the reason it is dark is the same reason the delete test failed.