What a number next door gets you

Before we can see the damage, one thing about networks has to be clear, and it is easiest to see with the brightness numbers from the last slide.

Picture a finished network that has already been trained. Feed it a pixel of brightness 128 and some number comes out the other side. Say it is 50. Now feed it 130 instead, and out comes 51.

That is not a coincidence, and nobody trained it on 130 specifically. It happens because of what a network physically is: a chain of arithmetic. The input number gets multiplied, added, and passed along. Feed in a number two bigger and every step downstream comes out a fraction different. Small change in, small change out.

So a network can never treat 130 as unrelated to 128. Nothing inside it is capable of that. The only thing it ever sees is the number, and 130 is nearly 128.

Everything else in this chapter depends on that. Whatever a network does with some number, it does roughly the same thing with the numbers on either side of it.

For brightness that is exactly what you want. 128 and 130 really are nearly the same grey, so anything true of one is almost certainly true of the other. The numbers were handed to us by the world, and their closeness was already meaningful before the network arrived.

But a network cannot switch this property on only when the numbers deserve it. It is not a judgement, it is just what arithmetic does. Hand it word IDs and it applies the same rule to 3 and 4, numbers we handed out in alphabetical order.

And you can probably already feel where that ends up. Sort a dictionary and volcano lands next to volleyball. Their IDs come out side by side, so the network is obliged to treat them as near-identical cases. Everything it learns about volcanoes, the erupting and the lava and the ash, it half believes about volleyball too.

What a number next door gets you

Before we can see the damage, one thing about networks has to be clear, and it is easiest to see with the brightness numbers from the last slide.

Picture a finished network that has already been trained. Feed it a pixel of brightness 128 and some number comes out the other side. Say it is 50. Now feed it 130 instead, and out comes 51.

So a network can never treat 130 as unrelated to 128. Nothing inside it is capable of that. The only thing it ever sees is the number, and 130 is nearly 128.

Everything else in this chapter depends on that. Whatever a network does with some number, it does roughly the same thing with the numbers on either side of it.