The neuron learns to answer in degrees

Before this version of training can trace a mistake back through those weights, one thing about the neuron has to change.

The correction rule from the last chapter worked because nothing was hidden. To fix a weight buried in a middle layer, training needs to know how much a small change there would alter the final answer.

The threshold cannot say. It reports on or off and nothing else. Nudge a weight so the total goes from 0.4 to 0.45 to 0.5, all under a threshold of 0.6, and the neuron answers no every time. It gives no sign of getting closer, right up to the moment it flips.

So this network replaces the hard threshold with something that bends rather than jumps. The turns any total into a value between 0 and 1, sliding smoothly from one to the other instead of flipping.

Now the neuron answers in degrees rather than yes or no. Barely, somewhat, almost certainly. Nudge a weight and the answer shifts a little, so a small improvement is visible as a small improvement.

If that sounds like the brain cell from the last chapter, it is the same idea reached by a different route. A brain cell says strongly by firing faster. An artificial neuron has no pulses and no clock, so it skips straight to the number the rate would have described.

The rule a neuron uses to turn its total into an outgoing signal is called an activation function. The threshold was one. The sigmoid is another. Backpropagation does not require sigmoid itself. It needs operations whose response to a small change can be calculated.

Which one you pick decides what an artificial neuron can say. The threshold could only ever answer 0 or 1. The sigmoid can answer anything between them, but never anything above 1. That ceiling belongs to the sigmoid, not to artificial neurons in general, and the next module shows why later networks abandoned it for a rule with no ceiling at all.

With that in place, the network can be asked the harder question: when the final answer is wrong, which of these weights caused it?