The Honest Limits

Every tool has a shape, and the shape decides what it is good for. We have spent two chapters learning to feed the model well. This chapter is the honest other half: the failures no amount of good feeding removes, because they live in the machine itself.

Language models are very good at producing fluent, plausible text. That is what they were trained to do, and they do it with a consistency that can feel like understanding. But fluent and plausible is not the same as accurate. The system has no way to check whether what it is producing is true. It only knows whether it fits the pattern.

This produces a particular kind of failure. Not random errors, not obviously broken outputs. Confident, well-formed, completely wrong answers. Invented sources that sound real. Plausible steps that lead nowhere. The surface is intact; the ground beneath it isn't.

These failures are not bugs to be patched. They come from the mechanism itself. Knowing that is the starting point for using these tools without being misled by them.