The learning rate

Look again at the rule that moved the weights:

weight change = error x unit x learning rate

Two of those three come from the wine. The error is however wrong the guess happened to be, and the unit is whatever the tasting produced. Neither is anyone's choice.

The third is entirely a choice. The learning rate was set to 0.1 back when the rule was worked out, which is why each correction moved a weight one tenth of the way the error called for. Nothing forced that tenth. It could have been half, or a hundredth.

Those are three ways of saying the same thing, and it helps to have them lined up. A learning rate of 0.1 is one tenth, which is 10 percent of the correction the error asked for. A rate of 1 would be the whole of it. A rate of 1.5 would be half as much again as the error called for, overshooting on purpose.

It is a number chosen before training, rather than learned from a bottle, and it decides how bold every correction is.

Anyone who has fought with a shower tap already knows this number.

The water runs cold and you want it warm. How far do you turn the tap? Not whether to turn it, and not which direction, both of those are obvious. Just how much, each time you adjust.

The learning rate is that amount of turn, and it fails in both directions.

Set it too small, say 0.001, a thousandth of the correction instead of a tenth. Every correction is a hair's width. This is the tap you nudge by a millimetre at a time: nothing goes wrong, the water does warm up, and you are still standing there shivering long after you wanted to be washed. The weights creep toward something sensible, but it takes thousands of bottles to get anywhere.

Set it larger and each correction is a bigger share of the error. Up to a point this is simply faster, the confident half turn that gets you to warm water quickly, and the model reaches good weights in fewer passes than a cautious tenth ever would.

Past a point it is not. Turn too hard and you shoot past comfortable into scalding, yank it back and land in freezing, and each panicked correction throws you further than the last. The water is never right, and you are getting further from a wash, not closer.

The weights do exactly this. Set the rate high enough and each correction overshoots by more than the last one missed by, so the guesses swing wider on every bottle instead of closing in. The total error grows pass after pass until the numbers stop meaning anything.

Where that point sits depends on the wines. It is not a fixed number to memorise, and a different model or a different set of bottles crosses it somewhere else entirely.

So there are settings of this one number that make a model actively get worse the longer you train it. Not stuck, not slow. Worse. Every additional bottle drives the weights further from anything useful, and no amount of extra training rescues it.

Somewhere between crawling and flying apart sits a range that works. Small enough not to overshoot, large enough to get somewhere in reasonable time.

There is no single learning rate that is right for every model. Builders use theory, previous runs, schedules, and experiments to narrow the choice, then watch the error to see whether training is stable.