Loading slide
Loading contents...
When people picture AI going badly, they usually picture one thing, a system far more capable than anything now, pursuing something we did not intend, and us unable to stop it.
That scenario should not be waved away. Some of the most serious researchers in the field consider it a genuine risk and have argued so carefully. Others, equally serious, think it rests on assumptions about future capability that may not hold. This journey has already said where that leaves us: the question is open, the disagreement is honest, and nobody gets to close it by asserting confidence. It remains open here too.
The argument about that scenario has taken up most of the room, and it has a specific feature. It is about a machine that does not exist yet.
There is a second way this goes badly, and it has the opposite shape.
It needs no new capability. It needs nothing to want anything. It is running now, and the last four slides described its parts.
Generation costs almost nothing, so the channel fills. Ranking rewards whatever performs, so the filling accelerates. Convincing text and images and voices are cheap, so establishing whether anything is real gets more expensive for everyone, and a genuine recording can be dismissed as fake by anyone who needs it to be. The models that could help you check are trained on the same sludge, and the human-made share of it keeps shrinking.
Nothing in that chain requires a breakthrough. Every link is a system behaving exactly as designed.
The end state is not dramatic. There is no moment. It is a slow rise in the cost of knowing anything, until most people stop paying it and settle for whatever is in front of them.
Both of these are worth taking seriously, and they are different kinds of claim. One is a contested prediction about systems we do not have. The other is a description of a process already underway, with published evidence for each step.
They also pull in different directions when it comes to what you would do about them. The first is largely a question for a small number of labs and governments. The second is about incentives, ranking rules, distribution, and who profits from volume, which is a much more ordinary sort of problem, and much less interesting to argue about.
Which may be part of why it gets less attention. Catastrophe is a better story than corrosion. Corrosion is the one you can already measure.