Loading slide
A neural network can know something and be completely unable to explain it.
Picture a face-recognition model. It can pick out a person across different lighting, angles, and ages, sometimes more reliably than a human can. But it cannot tell you what it looked for. There is no checklist it consulted, no chain of reasons you could read back. The knowledge lives in the geometry of billions of numbers, and that geometry does not translate into anything a person can inspect.
This is the sharp break from the old AI we met in the long-winter module. Back then, an expert system held rules a human had written down: if the ears are this shape and the nose this proportion, then it is probably a cat. Those rules were readable. You could argue with them, and you could fix the wrong one. A neural network has nothing like that. It is a black box, not because anyone hid the inside, but because distributed, learned knowledge simply has no readable inside. The understanding is in the whole, not in any piece you can point at.
Most of the time this does not bite. When the model is right, you do not care that you cannot see why. The trouble comes when it is wrong. You usually cannot find the mistake, because there is no single place the mistake lives. The error is spread across billions of weights, none of which can be opened up and questioned. That gap between what the model can do and what we can read out of it is not a flaw waiting for a patch. It is the price of this kind of learning.