Painting into a corner

A model can talk itself into a dead end, and it happens more than you would think.

Suppose it begins an answer with There are three main reasons for this. Seven words, perfectly sensible, and it has just made a promise. Notice that nothing about writing that sentence required knowing what the three reasons are. It only required that the sentence be a likely thing to say.

Now it has to deliver. Two of the reasons are solid. The third is not really there.

It cannot go back. That opening is not a draft, it is part of the input for every token that follows, and there is no mechanism anywhere for revising it. So the model does the only thing available to it and produces a third reason to fill the shape, thinner than the others, sometimes invented outright.

Every word along the way was a reasonable next word. It is the no-going-back rule that turns a series of reasonable steps into a corner you are standing in.

Which explains something people misread constantly. When a model drifts, or contradicts a line from earlier, or defends a weak claim it seems to have chosen at random, it is usually not confused the way a person would be. It made a locally sensible choice early and then had to live with it. There was never a draft to fix.