The snake eating itself

Now put the flood back next to the thing that made it, because they are connected in a way that took a few years to become visible.

Where did the training data come from? A very large collection of text written by people, scraped from the open internet. That was the whole premise. The model learned the patterns of human language by reading an enormous amount of human language.

Then the models were released, and the internet began filling with their output.

So the next training run scrapes a web that is no longer only people. Some fraction of it, growing and difficult to measure, is text that a model produced. The system is being fed its own output, one generation removed.

Here is why that matters mechanically rather than poetically.

A model learns the distribution of its training data: not just the common patterns but the whole spread, including the rare things at the edges. The unusual phrasing. The regional word. The odd case that appears twice in a million documents. When the model generates, it favours the likely, because that is what predicting the next token means. Its output is a slightly smoothed version of what it learned, with the tails thinned.

Train the next model on that, and it learns the smoothed version as if it were the truth. Then it smooths again.

Researchers tested this directly and published the result in Nature in 2024. Training models on successive generations of their own output produced two stages. First, the model drifts from the real distribution and starts getting the edges wrong. Then, later, the rare events stop appearing at all, permanently. They called it model collapse. Mixing real human data back in slowed the process; it did not prevent it.

The image is a copy of a copy of a copy. Each pass looks fine. Twenty passes in, the detail is gone and what remains is a blurry average, and nothing in the process ever announced that it had started going wrong.

Two things keep this honest.

This was demonstrated under deliberate experimental conditions, feeding output straight back in. Nobody is claiming the web collapses on a schedule. The labs know about it, and considerable effort now goes into filtering training data and tracking what is human-made.

But the underlying pressure is real and does not go away: human-written text became a scarce, valuable, and shrinking share of what is available to train on. Nobody has solved that, and the flood from the first two slides in this chapter is what is doing the diluting.

The system that needed the internet to exist is now steadily making the internet less useful for building the next one.