Loading slide
Loading contents...
A neural network can know something and be completely unable to explain it.
Picture a face-recognition model. It can pick out a person across different lighting, angles, and ages, sometimes more reliably than a human can. But it cannot tell you what it looked for. There is no checklist it consulted, no chain of reasons you could read back. The knowledge lives in the geometry of billions of numbers, and that geometry does not translate into anything a person can inspect.
This is the sharp break from the hand-written AI of the earlier decades. Back then, an expert system held rules a human had written down: if the ears are this shape and the nose this proportion, then it is probably a cat. Those rules were readable. You could argue with them, and you could fix the wrong one. A neural network has nothing like that. It is a black box, not because anyone hid the inside, but because nothing in there was ever written to be read. The understanding is in the whole, not in any piece you can point at.
Most of the time this does not bite. When the model is right, you do not care that you cannot see why. The trouble comes when it is wrong. You usually cannot find the mistake, because there is no single place the mistake lives. The error is spread across billions of weights, and no single one of them can be opened up and questioned. That gap between what the model can do and what anyone can read out of it is not a bug someone forgot to fix. It comes with this kind of learning, and closing any of it is hard, slow work.