Loading slide

What a Trained Model Actually Is

  1. 01What a Trained Model Actually Is
  2. 02Memory card
  3. 03A model is a file
  4. 04Two files
  5. 05How the two fit together
  6. 06A Piano
  7. 07The valuable file
  8. 08You connect to it
  9. 09Where does training data come from?
  10. 10Cleaning the data
  11. 11The network can't read
  12. 12Guess the next word
  13. 13Right or wrong
  14. 14One nudge, a billion times
  15. 15Take a breath
  16. 16Where your memories live
  17. 17A base model is not an assistant
  18. 18Why it's called pre-training
  19. 19The weights are the model
  20. 20The scale of it
  21. 21Why so many?
  22. 22What a weight actually is
  23. 23Everywhere and nowhere
  24. 24Using the model: inference
  25. 25The whole arrangement
  26. 26Reinforce your understanding
  27. 27Question: What is a model, really?
  28. 28Question: Learning from raw text
  29. 29Question: The model doesn't know what it doesn't know
  30. 30Quiz: answer
  31. 31The machine learned something real.
  32. 32Want to go deeper?
4 / 31
BackNext

Two files

That second piece is a program, written by humans, that defines the shape of the network. How many layers. How many neurons in each. Which neuron connects to which. It doesn't change during training and it doesn't change after.

This is worth slowing down on, because it's where two ideas can blur together. The neurons and their connections are the structure. The weights are the numbers that sit on those connections, one weight for each link, setting how strongly a signal passes along it. So the program lays out the empty wiring; the weights are what fill it in. The structure tells you how to read all those numbers, and which number belongs to which connection.

So every model is really two things (a slight simplification, but a useful one). The program is small, often just a few hundred lines of code, published openly for anyone to read. The weights are enormous, tens or hundreds of gigabytes of numbers, sometimes split across dozens of separate files.

When you run a model, the program loads the weights and uses them. The program is the shape. The weights are everything learned. Neither works without the other.

Citations(2)↓
  1. 1. docs.pytorch.org
  2. 2. github.com
Citations(2)↓
  1. 1. docs.pytorch.org
  2. 2. github.com