One pass is not much

However many rounds you pressed through, one full trip past the training set (in this case the eight bottles) has a name. It is called an epoch. Eight bottles, eight corrections, one epoch.

Our first epoch reduced the total error across all eight bottles from about 114 dollars down to 69.

That is a good day's work. Forty five dollars of error gone, in eight corrections, from a model that started out knowing nothing about wine.

Later epochs keep improving it, but each one gains less than the one before. The reason is that the bottles do not agree.

Each bottle pulls the weights toward its own answer, and none of them gets what it wants, because the next bottle immediately pulls them somewhere else. What one epoch produces is not the finished solution. It is a temporary compromise that nobody voted for.

Eight people share one speaker. Each has a volume they would like, and none of them says so out loud. They take turns, and each one is allowed a small nudge, a notch or two toward whatever they wanted. The person who likes it loud nudges up, the next person nudges back down, and the dial keeps moving. Nobody ever gets the volume they came for. Go round enough times and it settles somewhere every remaining nudge is tiny, because everyone is a little unhappy in ways that cancel out.

The bottles are those eight people, and the size of a nudge is the learning rate. The weights are what everyone is nudging.

The compromise is the point, though. Those weights are now shaped a little by all eight wines instead of perfectly by one, which is exactly what the greedy method could never manage.

If you went round more than once, you will have seen what happens next.

In this run, after the 45 dollars the first epoch removed, the second removes 20. The third removes 8, the fourth 4. Every one of them helps, and every one helps less than the last.

Many training runs fall quickly at first and then flatten, but not every run is this smooth. Error can wobble, stall, or rise when the learning rate or data causes trouble. Knowing when to stop is a real decision.

For now the useful question is a different one. The weights are creeping toward something, epoch after epoch.

Toward what?