Learning without updating

Here's something genuinely strange: a language model can appear to learn from examples in the prompt — without any training happening.

Give a model a few examples of a new task it's never seen before, and it will often generalize correctly to new instances. This is called in-context learning.

The model isn't updating its weights. The examples disappear the moment the context window is cleared. But within that context, the model uses the examples to do something that looks like inference about the task pattern.

Researchers don't fully understand why this works so well. The current best explanation is that during pretraining on enormous amounts of text, the model encountered countless examples of "here's a pattern, continue it" — and became extremely good at recognizing and extending patterns, including new ones provided at inference time.

In-context learning is one of the capabilities that distinguishes large models from small ones. It's also one of the things that makes prompting so powerful: you can give a model a new task with a handful of examples and get strong results, with no fine-tuning required.