The answer was already there

Both families train by the same trick, and it solves a problem that should have stopped all of this cold.

Look at what training has needed until now. ImageNet needed millions of images with a human-written label on each one. That labelling was years of work by thousands of people, and it was the expensive, slow, unavoidable price of admission.

Now consider the scale being described here. A huge slice of the internet. Books, articles, code. Nobody labelled that. Nobody could. There is not enough human labour on the planet to write an answer key for a trillion words.

So where do the correct answers come from?

They were already in the text.

Take any sentence from the pile. Cover the last word. You now have a question, the words before it, and the correct answer, the word you just covered. Uncover it and check.

Nobody wrote that answer key. The answer key was the text itself, with a piece temporarily hidden.

That is the whole trick, and it is why the internet could become training material. Every sentence ever written is already a completed exercise. The model does the exercise, then peeks at the answer that was sitting there the entire time.

This has a name: self-supervised learning. Supervised, because there genuinely is a correct answer to compare against, exactly as when a person wrote a label beside each photograph. Self, because the data supplied that answer itself rather than a person supplying it.