Loading slide
Look at the two pictures. Each one shows what a rule does to a number: the number you feed in runs along the bottom, the number that comes out rises up the side.
On the left is a plain multiply, "times 2." Its picture is a straight ramp. The more you put in, the more comes out, climbing forever with nothing to stop it. A straight line, never a bend. That is the collapse problem from a moment ago, drawn out: stack these and you just get one steeper straight line.
On the right is the rule the early networks actually chose. It was called
That cap is what creates the bend. A line that is not allowed to climb past 1 cannot keep running straight up. It is forced to level off near the top, and level off near the bottom too, with a steep rise in between. That is the shape on the right, a lazy letter S lying on its side. So "squash everything into 0 to 1" and "bend the line" are not two separate ideas. They are the same fact: the cap is the squash, and the squash is the bend.
For passing a signal forwards, that bend works nicely. The trouble is hiding in those two flat ends, and that is where the vanishing gradient comes from.