One iteration is many calculations

One training iteration sounds like one action. On a network with 100 million parameters it is a very large collection of smaller calculations, and training runs through that collection again for every batch of every epoch.

Every example in the batch moves through layer after layer. Within a layer, many values need the same kinds of multiplication and addition. The backward trip creates another large set of related calculations.

No single one of them is difficult. There are simply an enormous number of them, and most do not need to wait for the one beside them.

That is the shape of work a GPU was built for. A CPU, or central processing unit, usually has fewer powerful cores suited to varied instructions and branching work. CPUs can also run calculations in parallel, but GPUs devote much more of the chip to high-throughput arithmetic across many values.

The mathematical operation can be the same on either processor. In practice, the order of can differ, producing tiny numerical differences. The important change is throughput: a GPU can keep far more of the similar calculations under way at once.