Watch the note change

Use this sentence:

The dog ran because it heard the gate open.

The RNN receives these words one at a time. To make the carried numbers readable, imagine translating them into a short English note. The RNN itself never writes this note.

After The, the note says that something is about to be named. After dog, it says the sentence is about one dog. After ran, the RNN replaces it with a new note: one dog ran. When because arrives, the new note keeps the dog and the running, then adds that a reason is coming.

Each new note is made from two things: the note you already had and the new word. You do not keep a pile of old notes. You rewrite the one you are carrying.

When it arrives, the RNN combines that token with the note One dog ran, and a reason is coming. The earlier token dog can therefore affect what the RNN does with it.

Watch the note change

Use this sentence:

The dog ran because it heard the gate open.

The RNN receives these words one at a time. To make the carried numbers readable, imagine translating them into a short English note. The RNN itself never writes this note.

Each new note is made from two things: the note you already had and the new word. You do not keep a pile of old notes. You rewrite the one you are carrying.

When it arrives, the RNN combines that token with the note One dog ran, and a reason is coming. The earlier token dog can therefore affect what the RNN does with it.