Loading slide

The GPU Moment

  1. 01The GPU Moment
  2. 02One thing first
  3. 03Why training is slow
  4. 04What a GPU actually does
  5. 05The realization
  6. 06The scale this enabled
  7. 07Something unexpected
  8. 08Take a breath
  9. 09Data and compute
  10. 10Memory card
  11. 11Reinforce your understanding
  12. 12Question: Why GPUs and not faster CPUs?
  13. 13Question: Three ingredients
  14. 14Question: Why the field sped up
  15. 15Quiz: answer
  16. 16The hardware was ready. The data existed.
  17. 17Want to go deeper?
5 / 16
BackNext

The realization

The realization, when it came, felt almost obvious in retrospect.

Researchers at Toronto and NYU noticed that the core math of neural network training, multiplying large tables of numbers together, was exactly the kind of thing GPUs were built to do. In 2007, NVIDIA released , a platform that let programmers run general-purpose code directly on GPU hardware. Not just graphics. Anything.

Training that had taken weeks on a CPU took hours on a GPU. Some tasks were 10 to 50 times faster.

No new algorithm was invented. No new idea was discovered. Researchers just moved existing math to hardware that could run it in parallel, hardware that had been sitting in gaming computers and entertainment systems, used mostly to render explosions and racing games.

The speedup changed the size of experiments researchers could actually run.

Citations(2)↓
  1. 1. dl.acm.org
  2. 2. developer.nvidia.com