One helpful change

The sigmoid solved one problem and created another.

Take a neuron whose inputs and weights give a total of 2. The sigmoid turns that into 0.88. Now suppose training pushes the weights up so the total becomes 6, three times larger. The sigmoid turns 6 into 1.00.

The total jumped by 4. What the neuron passed on moved by 0.12.

That is the squeezing. The sigmoid could only ever return a value between 0 and 1, so once the total was already large, an enormous change going in made almost no difference coming out. And a correction travelling backwards has to pass through that same near-frozen step, arriving smaller than it left.

A simpler rule helped. ReLU, short for rectified linear unit, does two things. Any positive total passes straight through: feed it 2 and it returns 2, feed it 6 and it returns 6. The jump of 4 stays a jump of 4.

Any negative total becomes 0. On that negative side, ReLU's is also zero, so this route supplies no correction for that example. On the positive side, its local gradient is 1. This avoids the extra shrinking caused by a saturated sigmoid at that point, although other weights and operations can still weaken the full gradient.

ReLU reduced one source of vanishing gradients. It did not stop every gradient from shrinking, and a unit that remains on the negative side can stop learning.

ReLU was not the endpoint. Many modern language models use later activation functions. Llama 3 and DeepSeek-V3 use rather than plain ReLU. The lasting lesson is that the shape of an activation function changes how easily a network trains.