Why bother bending the number?

It seems pointless. The neuron does its work, then hands the result to a function that changes it. Why not skip that and pass the number straight on?

Here is the surprising answer. Without that step, stacking layers would buy you nothing at all.

Remember what a layer really does to a number: it multiplies and adds. Nothing more. So watch a single number travel through three layers. It starts as 10. The first layer multiplies by 2, giving 20. The next multiplies by 3, giving 60. The next by 5, giving 300.

Now here is the catch. Multiplying by 2, then 3, then 5 is the exact same thing as multiplying by 30 once. Ten goes in, three hundred comes out, either way. You could delete the two middle layers, set the survivor to "times 30," and get an identical answer. The middle layers added nothing.

That is the iron rule: a multiply followed by a multiply is still just one multiply. Stack a hundred such layers and the whole tower folds into a single "times something." A hundred layers, no smarter than one.

And a chain of multiplies can only ever do one kind of thing: scale a number smoothly up or down. It can never make a turn. Think of a rule like "a little salt improves the food, but a lot ruins it." Small is good, then it reverses and large is bad. Pure multiplying can never capture that reversal, that bend in the relationship. It only knows "more in, proportionally more out."

So how do you get a bend? Only one way. You need a step that is not another multiply, something that does a genuinely different kind of thing to the number. And because it is not a multiply, it cannot be folded into the multiplies around it.

That different step is the activation function. Drop one between the layers and they stop collapsing, because you cannot absorb a bend into a multiply. Now each layer truly adds something new, and the network can finally shape those turning, reversing, real-world relationships instead of one smooth slope.

So the function is not busywork. It is the only reason depth is worth building at all. Which makes the choice of which bend to use enormously important.

And here is the twist. The early networks all reached for one particular bend. It worked beautifully going forwards, but it was the very thing strangling the training signal on the way back. Let us look at it.