Loading slide

The Honest Limits

  1. 01The Honest Limits
  2. 02Memory card
  3. 03Why it sounds so sure
  4. 04The training data is the world
  5. 05What these tools are bad at
  6. 06Take a breath
  7. 07Reinforce your understanding
  8. 08Question: When to trust
  9. 09Question: Whose world is in the training data?
  10. 10Quiz: answer
  11. 11Knowing the limits is using them well
  12. 12Want to go deeper?
4 / 11
BackNext

The training data is the world

First, a word that means two different things. Back in the learning module, bias was a tiny technical knob: a single number that sets how easily a neuron fires. This slide is about the everyday meaning instead: a lopsided, unfair view of the world. Same word, completely unrelated idea. Keep them apart.

Here is the root of this second kind of bias, and it is not villainy, it is arithmetic. A model learns from text, and text is not a fair photograph of the world. It is a record of who got written down. People who write publicly, who have internet access, whose ideas circulate widely, fill page after page. Oral traditions, languages with little digitised text, communities further from the keyboard, leave far fainter traces, or none. The model learns the world as written, which is a particular, skewed slice of the world as it is.

And remember what learning does. The model does not store opinions; it soaks up statistical patterns from what it reads. So the skew comes back out as defaults. Ask it to imagine a nurse and a surgeon and watch which one it tends to make a woman. Notice which dialect it quietly "corrects" toward, which holidays it assumes, which names it pairs with which jobs. Nobody programmed those leanings. They are the shape of the training pile, pressed into the weights and poured back out.

This is why bias cannot simply be patched out after the fact. You can add corrective training to push back on specific cases, and builders do, but the overall make-up of the training data is the foundation underneath everything, and you cannot fully out-edit a foundation. A model trained on a lopsided record has learned a lopsided world.

What you can do is use it knowing the lean is there. Treat its defaults as the average of who got written down, not as neutral truth, and push against them when they do not fit the people or the situation in front of you.

Citations