Bias in AI starts long before the model
Most of it is baked in before anyone writes a prompt. The good news is you can usually see it coming.
Adrian Davies
I once gave a talk on bias in AI at an AWS meetup. Afterwards, the same question came up again and again. Which model is the fair one?
It is a good question. It is just the wrong place to look.
A model is a mirror
A model learns from what it is shown. Feed it a lopsided history and you get a lopsided model, only faster and more confident. It does not invent unfairness. It inherits it, then scales it.
Think of it like teaching someone to cook from one family's recipe book. They will get very good at that family's food. Ask them for anything else and they will guess.
The best known example is still one of the clearest. Amazon built an experimental tool to rank job applicants and trained it on ten years of CVs. It learned to mark people down for the word "women's", as in a women's chess club. The industry had hired mostly men, so the model decided men looked like good hires. Reuters reported in 2018 that the company had dropped it.
Faces tell the same story. The Gender Shades study, also from 2018, tested commercial facial analysis systems. Error rates were up to around 35 per cent for darker skinned women, and under 1 per cent for lighter skinned men. Nobody set out to build that. The photos the systems learned from just did not look like the world.
Where it creeps in
- What you collect. Who is missing from the data? Who is in it, but only in one kind of situation?
- What you call success. If "good" means whatever happened in the past, the model learns old habits.
- Who tests it. A team that all looks and sounds the same will miss the same things.
- Where it is used. A model that is fine in one place can be unfair in another.
Average accuracy hides the people it fails.
Four questions before anything ships
- Who is this for, and who is not in the data?
- What happens to a real person when it gets it wrong?
- Can a human see the decision and overturn it?
- Are we checking results for each group, not just the average?
None of these need a data science degree. They need one person in the room willing to ask, and a team willing to wait for the answer.
The model is the last step
Choosing a model matters. But by the time you get there, most of the decisions that make a system fair or unfair have already been made. By people, in spreadsheets and meeting rooms.
That is the hopeful bit. Bias is something you design out, not something you hope away.
People first. Data second. Model last.