The Problem with Sequences

Order matters in language. But so does distance.

To understand a sentence, you often need to connect words that are far apart. The pronoun refers to a noun three sentences back. The conclusion depends on a condition introduced in the opening paragraph. The meaning of a word shifts depending on context that arrived much earlier.

For a long time, the way machines handled language made those long-distance connections hard to form. Text was processed one piece at a time, left to right, with earlier context fading as new words arrived. The further back something was, the harder it was to hold onto.

This was not a flaw in the execution. It was a consequence of the approach. And it imposed a ceiling on how well these systems could handle anything that required holding context across a long stretch of text.