Letting people pick the better answer

So language models are trained the same way, by comparison, and that is the last of the three steps that build one.

The first two have already appeared. Pre-training on an enormous pile of text, which leaves a base model. Then fine-tuning on hand-written examples of good answers.

Here is the third. The model writes several answers to the same question. A person reads them and marks which is better. Not what the perfect answer would have been, just which of these is preferable. That judgment is far quicker to make than an ideal answer is to write, so it can be collected at a scale hand-written examples never reach.

Even so, there are never enough of them. Training needs millions of rounds of feedback, and people can supply thousands.

So the judgments are used to build something else first. A second model, entirely separate from the language model, is trained on those comparisons until it can look at an answer and predict the rating a person would have given it. A stand-in for a human reviewer, available in unlimited quantity.

The language model is then trained against that stand-in, for as many rounds as it takes.

So the model is not trained on what people wanted. It is trained on a prediction of what people wanted, and the two can come apart.

That third step is RLHF, reinforcement learning from human feedback. Systems also add rule-based feedback, safety filters, and instructions supplied by the application on top.

And it is where the thing you talk to gets its manner. Its willingness to answer, its hedging, its refusals, its tone. None of that was in the base model.

Letting people pick the better answer

So language models are trained the same way, by comparison, and that is the last of the three steps that build one.

The first two have already appeared. Pre-training on an enormous pile of text, which leaves a base model. Then fine-tuning on hand-written examples of good answers.

Even so, there are never enough of them. Training needs millions of rounds of feedback, and people can supply thousands.

The language model is then trained against that stand-in, for as many rounds as it takes.

So the model is not trained on what people wanted. It is trained on a prediction of what people wanted, and the two can come apart.

That third step is RLHF, reinforcement learning from human feedback. Systems also add rule-based feedback, safety filters, and instructions supplied by the application on top.

And it is where the thing you talk to gets its manner. Its willingness to answer, its hedging, its refusals, its tone. None of that was in the base model.