Loading slide

Loading contents...

[█████████░░░░░░░░░][██████████████░░░░░░░░░░░░░░]6 / 12
<back>next

It took several changes

There was no single switch that made deep networks trainable. Researchers improved several parts of the process.

Prepare one layer at a time. Researchers first prepared each layer, then trained the whole network. This was called layer-by-layer pretraining.

Choose better starting weights. Carefully chosen values made signals less likely to explode or fade immediately. This is weight initialisation.

Use less-squeezing rules. ReLU avoided sigmoid saturation on its positive side, although it introduced limits of its own.

Different networks needed different combinations. Depth became useful through a collection of improvements, not one magical fix.

# citations(3)↓
  1. [1]cs.toronto.edu
  2. [2]proceedings.mlr.press
  3. [3]proceedings.mlr.press