Loading slide
Loading contents...
There was no single switch that made deep networks trainable. Researchers improved several parts of the process.
Prepare one layer at a time. Researchers first prepared each layer, then trained the whole network. This was called layer-by-layer pretraining.
Choose better starting weights. Carefully chosen values made signals less likely to explode or fade immediately. This is weight initialisation.
Use less-squeezing rules. ReLU avoided sigmoid saturation on its positive side, although it introduced limits of its own.
Different networks needed different combinations. Depth became useful through a collection of improvements, not one magical fix.