Loading slide
Loading contents...
In 2014, one influential translation system used one recurrent network to compress a sentence into a fixed list of numbers, then another recurrent network to write the translation. This was called sequence to sequence learning.
It worked, but the fixed list had to carry the whole sentence. Long sentences exposed the bottleneck.
Later that year, Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio proposed a change. While writing each translated word, the system could look back across the earlier word states and give more weight to the useful ones.
That change was called attention.
It answered two limits of the older RNN approach.
1. An early clue had to survive every hidden-state update before a later word could use it.
2. Each word had to wait for the hidden state produced by the previous word.
What if <code>it</code> could reach straight back to <code>animal</code> instead?
The path between them would shrink from six steps to one. The clue would no longer need to survive every word in between.
Instead of receiving only one running summary, a word can receive information directly from other available words.
Within an attention layer, word positions no longer wait for one running hidden state to move down the sentence.
But direct access creates a new question. If it can reach every available word, how does it decide which ones matter?