Racist Algorithms: When Artificial Intelligence Discriminates
Discover how AI algorithms can discriminate and the underlying causes. Practical examples and solutions for more equitable artificial intelligence.
A simple counterfeit bill leads to the arrest of Robert Williams. An error in the Detroit police's facial recognition software triggers a Kafkaesque nightmare, revealing how algorithms can be more racist than the humans who program them.
It's January 2020 when Robert Williams is arrested in his front yard, in front of his wife and daughters. The charge? Stealing luxury watches from a store. The problem? Williams has nothing to do with that theft. He was framed by a facial recognition algorithm that confused his face with that of the real thief. After a night in jail and hours of interrogation, the police realize the mistake: Williams doesn't even remotely resemble the wanted person.
"Does this computer think all black people look alike?", Williams asks the detectives, showing them the suspect's photo. His question, seemingly ironic, hides a disturbing truth: police algorithms have become systematically racist.
The software that only sees white people
Williams' story is not an isolated case, but the predictable consequence of an algorithmic discrimination that pervades American security systems. Facial recognition, that technology we consider neutral and objective, has actually learned to "see" some people better than others.
The numbers are unequivocal: according to the MIT "Gender Shades" study, the error rate for light-skinned men is 0.8%, while for dark-skinned women it rises to 34.7%. A 40-fold gap that turns into ruined lives when these systems end up in the hands of the police.
Joy Buolamwini, the MIT researcher who discovered this problem, experienced it firsthand: the lab's facial recognition software couldn't recognize her face. "I literally had to wear a white mask to be detected," recounts Buolamwini, who founded the Algorithmic Justice League to fight these discriminations.
The America of algorithms that predict crime
But the problem goes far beyond facial recognition. In many American cities, increasingly sophisticated algorithms are being used not only to identify criminals, but to predict who will commit future crimes.
COMPAS (Correctional Offender Management Profiling for Alternative Sanctions) is one such system. Based on 137 questions, the algorithm assigns a risk score to inmates that influences judges' decisions on bail, sentencing, and parole.
An investigation by ProPublica revealed that COMPAS is systematically biased: Black people misclassified as "high risk" were almost twice as many as white people (45% vs. 23%). Conversely, white people misclassified as "low risk" were almost twice as many as Black people (48% vs. 28%).
The most emblematic case is that of Eric Loomis, arrested in 2013. The judge based his sentence partly on the COMPAS score, but Loomis was never able to find out how the algorithm arrived at that assessment: the company that created it considers the algorithm proprietary.
Chicago and the list of future criminals
Chicago took it a step further: the Strategic Subject Algorithm compiles a "heat list" of 1,500 people who, according to the algorithm, are most likely to commit crimes. Scores range from 0 (low risk) to 500 (high risk).
The list allows police to constantly monitor the movements of these individuals and intervene if they do "something suspicious." The problem? 84% of the people on Los Angeles's list (which uses a similar system) are African American or Latino, in a city where African Americans make up only 9% of the population.
Even more disturbing: about half of these people had never been arrested for gun possession and 10% had never had any contact with the police. They are in the database solely due to an algorithmic prediction. This raises profound questions about predictive surveillance and its risks.
PredPol: when mathematics becomes racist
PredPol, used in over 60 American police departments, promises to predict where crimes will occur with near-scientific precision. The algorithm analyzes historical crime data and indicates "hot spots" where patrols should be concentrated.
The problem is that PredPol ends up perpetuating and amplifying existing discrimination. How does it work? The algorithm learns from historical arrest data, but we know that police arrest more people in neighborhoods with ethnic minorities. This leads the algorithm to direct more and more patrols to those areas, generating more arrests, which in turn "confirm" the algorithm's prediction.
This is what experts call a "discriminatory feedback loop": the algorithm replicates and amplifies existing biases, turning them into self-fulfilling prophecies.
In spatially segregated cities like those in America, even a home address becomes an indicator of ethnicity and income. PredPol can therefore learn to be racist without ever explicitly using categories like race or social class.
The Hidden Bias in Data
The root problem is that algorithms learn from our data, and our data reflects society's inequalities. As one expert explains: "If you teach a child for decades that people of color should be treated poorly, that child will grow up following these teachings. The same is true for algorithms."
Crime data is not neutral: it reflects human decisions about who to arrest, where to patrol, what to consider suspicious. When police check certain neighborhoods more often, they will obviously find more crimes in those areas, even if the actual crime rate is similar everywhere.
One study showed that African American and Hispanic males between 14 and 24 years old represent only 5% of the American population, but undergo 41% of police stops. 90% of these stops end with a release due to innocence. But in the meantime, these young people end up in databases as "police contacts," feeding predictive algorithms. The ACLU has extensively documented how these practices fuel vicious cycles of discrimination.
Italy and KeyCrime: A Different Approach?
Italy also has its own predictive policing algorithm: KeyCrime, developed by Mario Venturi and used by the Milan Police Headquarters. The results seem positive: robberies at supermarkets, shops, and pharmacies have dropped by 57%.
Unlike American software, KeyCrime uses much more personal data, focusing on individual people rather than just geographic areas. As Venturi himself explains: "The meticulous collection of this information is aimed at identifying traits that characterize the criminal event, and therefore the person who committed it."
However, this very invasive approach raises questions about privacy and surveillance. If the training data contains biases, KeyCrime also risks perpetuating them.
The Legal Consequences of Algorithmic Discrimination
The problem of algorithmic discrimination is finally reaching courtrooms. In 2021, the Court of Bologna convicted Deliveroo's "Frank" algorithm for discrimination against riders, setting an important precedent: for the first time, an algorithm was found legally responsible.
In the United States, class-action lawsuits like the one against Workday are multiplying. The company is accused of using algorithms that discriminate against candidates based on race, age, and disability in hiring processes. This highlights how AI in the future of work can create new forms of discrimination.
New York has passed a groundbreaking ordinance: employers cannot use "automated tools for employment decisions" unless they have passed a bias audit within the last year. It is the first law of its kind in the United States.
Europe Responds with the AI Act
The European Union has responded with the AI Act, the world's first comprehensive regulation on artificial intelligence. The new rules include specific provisions against algorithmic discrimination:
- Article 5: prohibits the use of AI that could create unjustified discrimination, especially in decision-making processes concerning individuals
- Article 10: requires that data used to train algorithms be free of bias and representative of the population
For facial recognition systems in public spaces, the AI Act requires "more rigorous assessment procedures" and "authorizations that address specific risks." An important step towards regulating artificial intelligence.
The Human Cost of the Racist Algorithm
Behind every statistic, there is a human story. Robert Williams had to explain to his daughters why their father had been arrested. He lost a day of work, suffered the humiliation of a public arrest, and had to face the anxiety of a criminal proceeding.
Kylese Perryman, the young man represented by the ACLU of Minnesota, lived through a similar nightmare: arrested and detained based solely on a mistaken facial recognition identification.
These are not acceptable "system errors" or "false positives." They are lives ruined by algorithms that have learned our biases and apply them with the ruthless efficiency of machines.
How to Stop Algorithmic Discrimination
The solution is not to eliminate algorithms, but to make them fairer. Experts suggest several strategies:
Diversify development teams: Include people from diverse backgrounds to catch biases that might otherwise go unnoticed.
Improve datasets: Ensure training data actually represents the entire population, not just dominant groups.
Independent audits: Regular external checks to identify emerging discrimination.
Algorithmic transparency: Make decision-making criteria understandable, at least to those affected by them.
Human oversight: Always maintain human control over critical decisions, especially in the criminal justice system.
Control feedback loops: Break the vicious cycles that amplify existing biases.
The Future of Algorithmic Justice
Some companies have already taken a stand. Following the George Floyd protests, IBM completely withdrew its facial recognition technology, stating it would "no longer offer facial recognition technology for mass surveillance and racial profiling to police departments."
Microsoft and Amazon temporarily suspended sales of these systems to law enforcement, pending clearer regulation.
But the problem goes beyond individual companies. As Joy Buolamwini points out: "It's not just about fixing flawed algorithms, but about addressing the structural problems those flaws highlight."
Towards a Fairer Artificial Intelligence
Artificial intelligence is not neutral: it is a mirror that reflects the biases, inequalities, and priorities of the society that creates it. Racist algorithms are not a system bug, but a feature that emerges from the discriminatory data we train them on.
The challenge is not to create a "perfectly neutral" AI – a likely unattainable goal – but to develop systems that actively promote equity and justice. This requires:
- Problem Recognition: Acknowledge that algorithmic discrimination exists and is widespread
- Shared Responsibility: Programmers, companies, institutions, and civil society must work together
- Democratic Oversight: Citizens must have a say in how these systems are used
- Restorative Justice: Those who have suffered algorithmic discrimination must be able to obtain redress
As Robert Williams, the man wrongfully arrested in Detroit, says: "If the technology can't tell the difference between one Black man and another, maybe it shouldn't be used by the police."
This is a lesson that goes beyond technology: in a democratic society, tools of power must be fair for everyone, or they should not exist.
Algorithmic discrimination is not a technological inevitability. It is a human choice that we can and must change. As highlighted by the principles of AI ethics, we must build systems that respect human dignity and promote equity for all.
To stay updated on these crucial issues, organizations like the AI Now Institute and the Partnership on AI regularly publish research and guidelines for the responsible development of artificial intelligence.