Take a breath

The names matter less than the mechanism. Here is the whole machine in plain language.

An RNN reads word by word, left to right, while carrying one fixed-size summary. Read a word, update the summary, move on. In a long passage, an early clue must survive rewrite after rewrite until the network needs it.

It is like reading through a keyhole. The RNN receives the current word. For every earlier word, it only has whatever survived in the hidden state.

Distance makes this harder because each extra word adds another rewrite that an early clue must survive.