Guess the next word

Here is the task: take a sequence of text from the dataset, hide the last piece, and ask the model to guess what it is.

Imagine we the text:

"The sky is full of stars."

And then we hide the last word:

"The sky is full of ___"

The model has to guess what comes next. It doesn't pick from a short list. It produces a probability score for every word it knows. Every single one. That's typically somewhere around 100,000 options, though as we'll see later, they aren't quite words. More like fragments of words. But the scale is what matters: tens of thousands of options, every single time, for every single guess.

Here is the part that surprises people. At the very beginning, the model has no idea. Remember, the weights start out as random numbers. So the first time it sees this blank, it spreads its guess almost evenly across everything it knows. "stars" gets about the same score as "banana", "running", or "the", as you can see on the right. Every option lands near the same tiny percentage. The guess is pure noise.

If this feels familiar, it should. This is the same engine you watched learn to tell dogs from cats: make a guess, see how wrong it was, nudge the weights, try again. The loop hasn't changed at all. What's changed is the target. Back then the guess was one of two options. Here it is one of a hundred thousand, made fresh for every single word.

The right answer was "stars." The model didn't know that. So how does a pile of random numbers ever learn to put "stars" on top?