A strange kind of knowledge

Neural networks know but can't explain.

A network can recognize faces reliably, across lighting conditions and angles, in ways that outperform most humans. But it cannot describe what it's looking for. There's no list of features it checks. No chain of reasoning that can be read or audited. The knowledge lives in the geometry of billions of numbers, and that geometry doesn't translate back into anything a human can inspect.

This is different from expert systems, where the knowledge was written down as rules and could at least be read and questioned. A neural network is a black box — not because anyone designed it to be, but because that's what distributed, learned knowledge looks like.

The opacity is mostly fine when the model is right. It becomes a problem when the model is wrong and you need to understand why.