Loading slide

The Problem with Sequences

  1. 01The Problem with Sequences
  2. 02Memory card
  3. 03Reading through a keyhole
  4. 04The running summary
  5. 05The bottleneck
  6. 06Take a breath
  7. 07The distance problem
  8. 08Memory card
  9. 09The same problem, wearing a new coat
  10. 10What a solution would have to do
  11. 11Reinforce your understanding
  12. 12Question: The hidden state
  13. 13Question: Long-range dependencies
  14. 14Question: The same problem twice
  15. 15Quiz: answer
  16. 16What to carry out of this chapter
  17. 17Want to go deeper?
10 / 16
BackNext

What a solution would have to do

Let's draw the bar plainly, because the whole rest of the module is the field clearing it.

Picture the sentence as a row of words. In the old way, the only paths run between neighbours: word one to word two, word two to word three, like stepping stones in a single line. To get from the first word to the last, you must touch every stone in between, and the message fades with each step.

Now imagine throwing a direct line between any two words, however far apart. "It" reaches straight back to "animal" in one hop, no stops, no fading. And not just that one pair. Every word gets a direct line to every other word, all at once. The sentence stops being a single thread you walk down and becomes a web where everything can touch everything.

Picture two changes happening together. First, the distance problem vanishes: if "it" can reach "animal" in a single hop, it makes no difference whether "animal" is two words back or two hundred. There is no chain left to fade along. Second, and this matters more than it first appears, there is no longer any reason to read the words strictly in order. If every word can see every other word directly, you can look at them all at the same time instead of trudging through them one by one. We will come back to why that second change was such a big deal, because it turned out to matter enormously for training these models.

That web, every word directly connected to every other word, weighing how much each one matters, is the idea called attention. It is the answer to this entire chapter, and building it up properly is the next one.

Citations(2)↓
  1. 1. arxiv.org
  2. 2. research.google
Citations(2)↓
  1. 1. arxiv.org
  2. 2. research.google