Turning scores into chances

Every score goes through the same small piece of arithmetic, a step called , and it does two things.

It makes every number positive. -2.7 comes out as a very small positive quantity, not a negative share. Nothing can end up owing probability.

It pulls the gaps apart. jar at 3.6 and lid at 2.4 are only 1.2 apart to start with, which sounds like a near tie. After softmax, jar is worth more than three times what lid is worth.

Then every value is divided by the total, so the whole vocabulary adds up to exactly 100 percent.

Those four tokens come out as jar 73 percent, lid 22, box 4, and ievable a little over one tenth of one percent.

Look at what happened to that gap. 1.2 between the top two scores became 51 percentage points between their chances. A modest lead going in is a commanding lead coming out, and that is not a side effect. It is what softmax is for.

Which is worth testing before going further, because those percentages have a consequence that sounds wrong.

Turning scores into chances

Every score goes through the same small piece of arithmetic, a step called , and it does two things.

It makes every number positive. -2.7 comes out as a very small positive quantity, not a negative share. Nothing can end up owing probability.

Then every value is divided by the total, so the whole vocabulary adds up to exactly 100 percent.

Those four tokens come out as jar 73 percent, lid 22, box 4, and ievable a little over one tenth of one percent.

Which is worth testing before going further, because those percentages have a consequence that sounds wrong.