Loading slide
The fix is almost comically simple. So simple it seemed, at first, like it couldn't possibly be the answer.
Drop the S-curve. Replace it with one plain rule: if the number is positive, pass it straight through, unchanged. If it is negative, output zero. That is the whole thing, and it has a name,
Now, you might object: "pass it through unchanged" sounds like doing nothing, like multiplying by 1. And did we not just say a chain of plain multiplies collapses into one? So how is this any different?
Here is the trick, and it is the word if. ReLU is not one rule, it is two rules with a corner between them. Flat at zero for every negative input, then it turns and climbs straight up for every positive one. Draw it and you get an elbow. That corner is a bend, a real one, just like the sigmoid's curve was a bend. So ReLU does not collapse. Stack many of them and you get many corners, an intricate folded shape no single multiply could make.
But look at the difference from sigmoid. The sigmoid bent by going flat at its ends, and flat is where the signal died, multiplied down by 0.2 again and again. ReLU's bend never flattens on the live side. For a positive number the line stays at full steepness, so the backward signal passes through at full strength, times one, not times a fraction. It either continues undimmed or, on the zero side, stops cleanly.
So ReLU threads the needle. It keeps a bend, so depth still means something. But it throws away the flat parts that were strangling the signal. One tiny change to the step inside each neuron, and deep networks became trainable at last.