Where did the rule go?

In a program built from written rules, you can point at one of them. Someone typed it:

If these parts are dark, call the shape a square.

Nobody wrote the complete sorting rule for the Perceptron. People still chose the examples, the input measurements, and the labels "square" and "dark." Training found weights that combined those inputs into the requested distinction.

So where did the rule go?

It was no longer a sentence. The learned distinction was spread across the weights.