Confident and wrong

A model that stores knowledge in distributed weights has no way to flag its own uncertainty.

A database can tell you "this record doesn't exist." A language model can't check whether a fact is in its weights before generating it — it just generates what's most probable. And what's most probable often sounds correct, because the training data contained mostly correct information.

This produces a specific failure mode: hallucination. The model generates text that sounds authoritative and is completely false. It might invent citations to papers that don't exist. It might describe a historical event that never happened, in accurate-sounding detail. It might give a confident answer about a law that doesn't say what the model claims.

The model isn't lying. It has no concept of truth versus falsehood. It's predicting what would plausibly follow from the prompt — and sometimes what would plausibly follow is wrong.

This is structural. It's not a bug that will be fixed in the next version. Understanding it changes how you should use these tools.

<!-- TODO: a short, vivid example of a hallucinated citation or fake-but-plausible "fact" would make this concrete without being alarmist -->