The lesson that never arrives

Line the words up by their IDs and you get a street. dog at number 3, dollar next door at 4, puppy thousands of doors away at 6,000.

Now train a model on this street. It reads a great deal of text and learns something true: after dog, the word barked often comes next. Its arithmetic gets tuned so that an input of 3 leans toward barked.

Ask what that tuning did to number 4. From the last slide, we already know it cannot leave 4 alone. A model that leans toward barked at 3 leans that way at 4 as well, because 4 is a hair away from 3 and the arithmetic runs smoothly between them.

Number 4 is dollar, so the model now half expects dollars to bark, having never been shown a sentence claiming that. Meanwhile the words that should have benefited get nothing. puppy sits at 6,000. hound, terrier and spaniel are scattered wherever the alphabet dropped them, none of them near 3, each having to meet barked in the text and learn the pattern from nothing.

Follow one of those all the way out and it ends somewhere a person would notice.

Go back to volcano and volleyball, sitting next door to each other. Someone types "the volleyball match was interrupted when the ground began to". A model with honest numbers reaches for shake. This one carries a faint pull toward erupt, borrowed from a neighbour it was never related to, and every so often that pull wins.

Nothing malfunctioned to produce that. The model applied the one rule it has, that close numbers mean close cases, to numbers that were never close in meaning.