The disturbing rise of racist robots

One of us: This robot called Sophia was granted Saudi Arabian citizenship in 2017. © Lightrocket

Are robots racist? AI computers are showing sexist and racist bias. A growing number of experts fear that we have created AI in our own worst image, and it could have awful consequences.

You’ve probably seen the images popping up on social media. Pictures of your friends and acquaintances, their faces framed by green boxes and strange labels. It’s all down to ImageNet Roulette, an artificial intelligence (AI) craze sweeping the internet.

It’s a simple idea. Upload a selfie to an AI computer, and it will categorise you based on its knowledge of over 14 million photographs from across the internet.

Some of the labels it gives out are flattering — who doesn’t want to recognised as “leader” or “president”? — and others are less so: “swot”, “nerd”, “dweeb”.

But some are disturbing. Chinese-Jewish writer Julia Carrie Wong wrote in The Guardian that the programme called her a racist slur. Many more people of colour have reported offensive labels.

You might think something went horribly wrong but, in fact, ImageNet Roulette is working exactly as its creators intended.

“We want to shed light on what happens when technical systems are trained on problematic training data,” they wrote.

AI computers are fed data to help them learn and perform their tasks better, but that data comes from a world peppered with prejudice and discrimination. As they learn the data, robots are also learning to mimic our biases.

There are other examples: a LinkedIn search that prioritised male profiles; a ChatBot fired out anti-semitic messages after learning to “speak” from Twitter.

“People expected AI to be unbiased; that’s just wrong. If the underlying data reflects stereotypes, or if you train AI from human culture, you will find these things,” says Joanna Bryson, an AI researcher.

“It’s racism in, racism out,” is how activist Hamid Khan sums up the problem. And it is already having disturbing consequences.

A US court started using a computer programme to assess how likely prisoners were to re-offend. Last year, a study found that the system was much more likely to mistakenly label black defendants as likely reoffenders. This happened to black prisoners twice as much as it did to white.

But why? The research suggested that the data (arrest records, postcodes, social connections and income) the programme was fed can all reflect problematic stereotypes, and further ingrain them.

“Police can say: ‘We’re not being biased, we’re just doing what the math tells us’,” warns Kristian Lum, who works for the Human Rights Data Analysis Group.

This year, tech firms are expected to spend $35.8 billion (£31.8 billion) on AI development. What kind of future are we heading towards?

Heart of darkness

Are robots racist? For all the talk of machine learning, the bias in AI computers proves that they cannot learn like humans: they don’t understand morality, or institutional discrimination, or other unequal forces that shape society. We must beware of giving power and knowledge to something that is so hard and coldly rational, without a shred of human warmth.

But surely it is wrong to say that it’s the robots that are racist? They are a blank slate, ruled by logic. Of course they can’t be biased. They are taught to be so by us. Corners of the internet teem with hatred; unemployment and income figures show constant racial inequality. AI is mimicking our worst instincts: it is we who need to reform.

You Decide

  1. Should we stop developing artificial intelligence?
  2. Will humans ever stop being prejudiced?


  1. Write five commandments or rules of your own about how AI should be used.
  2. Write a short story set in a world where AI is much more developed and widespread than it is now. Is it a utopia or a dystopia?

Some People Say...

“In general, when it comes to AI, many of us subconsciously cling to the selfish notion that humanity is the endpoint of evolution.”

Steve Jurvetson, US businessman

What do you think?

Q & A

What do we know?
Machine learning is the area of AI that seeks to teach robots how to learn and change their behaviour like human beings. It is how Netflix knows what films it should recommend to you, for example. It also means that computers can perform tasks on their own without specific instructions.
What do we not know?
How we can make AI less biased. Many private companies who make the algorithms are secretive about how they work, fearing their rivals could copy the technology. This makes it hard to detect when AI bias is at play, and alter the programmes to rectify it. Sandra Wachter, a law scholar at the Alan Turing Institute at Oxford University, has called for a “right to explanation”, which would require more transparency about AI algorithms.

Word Watch

Someone who you have met a few times, but don’t know that well.
Artificial intelligence
Developing computers that can perform tasks that normally require human intelligence, such as visual perception and speech recognition.
Unfair prejudices in a particular direction.
Oversimplified, usually negative ideas of a person or group.
He works for a group campaigning against LA Police Department’s use of AI.
To firmly fix something.
Many private companies who make the algorithms are secretive about how they work, fearing their rivals could copy the technology. This makes it hard to detect and erase AI bias.

PDF Download

Please click on "Print view" at the top of the page to see a print friendly version of the article.