The clue must survive every step

Return to this sentence:

The animal didn't cross the street because it was too tired.

When the RNN reaches it, it does not receive the original word animal again. It only receives the current hidden state.

The clue from animal must have survived the hidden-state update after didn't, then further updates after each of cross, the, street, and because.

That is five rewrites before the clue is needed by it. A longer passage may demand fifty or five hundred.

Every rewrite is another chance for the clue to weaken or be overwritten. Nearby words have a short path between them. Distant words have a long one.

When understanding one word depends on information much earlier in the sequence, that relationship is called a .

The clue must survive every step

Return to this sentence:

The animal didn't cross the street because it was too tired.

When the RNN reaches it, it does not receive the original word animal again. It only receives the current hidden state.

The clue from animal must have survived the hidden-state update after didn't, then further updates after each of cross, the, street, and because.

That is five rewrites before the clue is needed by it. A longer passage may demand fifty or five hundred.

Every rewrite is another chance for the clue to weaken or be overwritten. Nearby words have a short path between them. Distant words have a long one.

When understanding one word depends on information much earlier in the sequence, that relationship is called a .