What Understanding Actually Is

At some point, impressive performance stops being enough.

You can watch a machine do something remarkable and still find yourself asking a quieter question: is there anything home? Does it understand what it's doing, or is it producing outputs that look meaningful without anything behind them?

This is not a question you can settle by watching more closely. Better performance doesn't answer it. More impressive results don't answer it. The question is about what's happening on the inside, and the outside gives you no direct view of that.

That gap, between behavior and understanding, is one of the oldest puzzles in the study of mind. But it becomes newly sharp when the thing producing the behavior is a machine. Because then you have to ask not just whether understanding is present, but what understanding even is. What would have to be there for it to count?

This chapter sits with that question. It doesn't resolve it. But it makes it harder to look away from.