Loading slide
Here is the sharper problem, and it is worth slowing right down for, because attention is the answer to exactly this.
Take a sentence you understand without effort:
The animal didn't cross the street because it was too tired.
What does "it" point to? The animal, obviously, not the street. A street does not get tired. You connected "it" back to "animal" instantly, without even noticing you did it.
Now ask the RNN to make the same connection. "It" sits near the end. "Animal" sits near the start. And remember how the RNN works: it only ever passes its little summary forward, one word at a time. So for anything about "animal" to reach "it," it cannot jump straight there. It has to ride the summary forward through every word in between: through "didn't," "cross," "the," "street," "because," surviving each read-mix-replace along the way.
Think of it as a message whispered down a long line of people, one of the exact pictures from the deep-learning module. Each person passes it to the next, and each handoff blurs it a little. Over two or three people, the message arrives fine. Over a dozen, it has degraded into mush. By the time "animal" reaches "it," nine words later, there may be almost nothing left of it.
So the further apart two related words sit, the weaker the thread between them, until it frays completely. Short connections, the RNN handles well. Long ones, the kind that hold real sentences together, are exactly where it falls apart. And language is full of long ones: a pronoun pointing back across a paragraph, a verb that depends on a subject from sentences ago.
If this fading-down-a-line feeling reminds you of something specific from the last module, hold that thought. The next slide names it, because it is not a coincidence.