Loading slide
A model only knows what was in its training. That sounds too obvious to be worth saying, until you follow it through.
If something was not in the training data, the model has no knowledge of it. But it will not tell you so. Asked about something beyond its experience, it does not stop and say "I have not seen this." It produces the most plausible-sounding answer it can, because producing the most plausible-sounding answer is the only thing it ever learned to do. There is no part of the model that checks "is this inside what I actually know?" before it speaks.
Think of a doctor trained in one country, seeing one population of patients for an entire career. Within that world she might be superb. Ask her about a disease common somewhere she has never practised, and she may answer confidently and be wrong, not from carelessness, but because that pattern was simply never in her experience. The gap is invisible to her.
That is the model, with one difference that makes it sharper. The model's knowledge has a hard edge, the boundary of what it was trained on, and the model has no sense of where that edge is. It cannot feel itself crossing from the things it knows into the things it is inventing. This is why fluency is such a poor guide to accuracy: the answer beyond the edge sounds exactly as smooth as the answer well inside it. Knowing when to trust a model starts with a rough sense of what it was, and was not, trained on.