The Ethical Dimension of AI in Judicial Surveillance Processes: Justice or Automated Bias?
If an algorithm decides you are "at risk of criminality" just because you live in a certain neighborhood, is that justice or mathematical prejudice? AI is enter
Imagine being arrested. You haven't committed any recent crime, but software has calculated there's an 85% probability you will commit one in the next 48 hours. The police knock on your door for a "preventive" check. Or imagine being before a judge to request parole. The judge looks at a screen, sees a red score generated by a proprietary algorithm (whose code no one knows) and denies the request. Not because you did something wrong in prison, but because historical data says people "like you" (same neighborhood, same ethnicity, same income) tend to reoffend.
This scenario is not the plot of Minority Report. It is the daily reality in many jurisdictions that use Predictive Policing and Risk Assessment Tools. Artificial Intelligence promises to make justice more efficient, faster, and objective. But what happens if the algorithm is not an impartial judge, but a mirror that amplifies the darkest prejudices of our society?
In this article, we will explore the ethical dimension of algorithmic surveillance, analyzing how bias in historical data transforms into systemic discrimination, what the new European AI Act provides to stop this drift, and whether it is possible to build an AI that serves justice without trampling on fundamental rights.
1. The Myth of Neutrality: Predictive Policing and Bias
The idea behind predictive surveillance is seductive: use data to allocate police resources where they are most needed. However, as analyzed in our in-depth look at predictive surveillance and predictive policing, there is a fundamental logical error: AI does not predict crime, it predicts police activity.
The COMPAS Case: Racism in Code
The COMPAS software, used in US courts to estimate recidivism risk, has become the symbol of ethical failure. A foundational study cited by OxJournal demonstrated that the algorithm had a false positive rate (people labeled as high risk who did not commit new crimes) of 45% for African Americans, compared to 23% for white people. The algorithm did not explicitly use the variable "race." It used proxies like zip code, income, friendships, and prior arrests. But in a society where minorities are historically over-policed, using historical arrest data means teaching the AI that "being black in that neighborhood" equates to "being a criminal." As denounced by the NAACP, this creates a devastating feedback loop: police are sent to "high-risk" neighborhoods, arrest more people for minor offenses (which elsewhere would be ignored), the arrest data feeds the algorithm, which confirms the neighborhood is "high-risk." It is a self-fulfilling prophecy automated.
Historical Bias vs. Algorithmic Fairness
The problem is not that the algorithm is "bad." It's that it's "stupid." It learns from the data we give it. If the data reflects decades of systemic discrimination, the AI will do nothing but automate and speed up that discrimination, giving it a veneer of scientific objectivity ("The computer says so"). To delve deeper into how human prejudices infect code, we refer you to our article on algorithmic bias and invisible discrimination.
2. The Regulatory Wall: The AI Act and European Bans
While the algorithmic Wild West continues in the USA, Europe has drawn a red line. The new AI Act represents the world's most ambitious attempt to regulate the use of AI in justice.
High-Risk Systems and Absolute Bans
Article 5 of the AI Act, as explained by Artificial Intelligence Act EU, explicitly prohibits some practices considered "unacceptable" for fundamental rights:
- Social Scoring: Prohibited to use AI to assess a person's trustworthiness based on social behavior.
- Predictive Policing based on Profiling: Prohibited the use of systems that assess a person's risk of committing crimes based exclusively on profiling or personality traits, without concrete facts.
- Emotion Recognition: Prohibited to infer emotions in law enforcement contexts (e.g., during an interrogation), because the science behind it is unreliable and the risk of abuse is extremely high.
Furthermore, AI systems used to assist judges or evaluate evidence are classified as "High-Risk". This means they must comply with rigorous obligations: high-quality datasets (to minimize bias), total transparency (no "black box"), mandatory human supervision, and logging of all records for future audits.
The Council of Europe's Ethical Charter
Not just laws, but principles. The European Ethical Charter on the use of AI in judicial systems establishes the principle of "user control": the judge must always be able to deviate from the algorithm's decision and must be able to explain its logic to the parties. "Automated justice" without a human face is considered incompatible with the rule of law.
3. Judicial Ethics: The Judge in the Machine Age
The introduction of AI in courts raises deep deontological questions for magistrates and lawyers.
The "Black Box" Risk
If a judge uses software to decide a sentence, but does not know how the software arrived at that conclusion (because it is covered by industrial secrecy), they are abdicating their duty to provide reasoning. How can a defendant defend themselves against an accusation generated by a black box? As highlighted by the National Center for State Courts (NCSC), the uncritical use of opaque tools violates the principle of "due process."
Automation Bias in Courts
Then there is the psychological risk. Studies cited by the OECD show that humans tend to blindly trust computer suggestions ("Automation Bias"). If the software says "High Risk," a tired or overworked judge might tend to confirm that assessment for safety, turning the algorithmic suggestion into a de facto sentence. We analyzed whether AI could ever replace the robe in our provocative article: Will AI replace the judge? Automated justice pros and cons.
4. Beyond Criticism: An AI for Fair Justice?
Is it possible to use AI ethically in justice? Some experts say yes, provided we change the paradigm.
Auditing and Transparency
Organizations like Tranquility AI suggest introducing mandatory independent algorithmic audits. Before being used in a court, software should be "stress-tested" to verify if it discriminates against minorities, just like a drug is tested before being marketed.
AI for Defense, Not Just Prosecution
AI can also be a tool of guarantee. It can analyze millions of pages of documents to find exculpatory evidence that a human lawyer might miss. It can monitor judges' sentences to detect their human biases (e.g., judges who sentence more harshly before lunch) and flag them for correction. The goal must not be "predictive policing," but "augmented justice," where technology helps reduce human error, not systematize it.
5. Mass Surveillance and Fundamental Rights
Finally, we cannot ignore the broader context. The judicial use of AI fits into an increasingly pervasive ecosystem of mass surveillance. As discussed in our article on mass surveillance and privacy defense, facial recognition and behavioral analysis technologies used for "security" progressively erode the space of individual freedom. If our every movement is tracked and evaluated by an algorithm looking for anomalies, are we still free citizens or are we all "suspects awaiting judgment"?
Frequently Asked Questions
Is Predictive Policing legal in Europe? The AI Act bans systems based exclusively on profiling or personal characteristics. However, the use of analytical software to map crime "hotspots" (places, not people) is still permitted, provided there are rigorous safeguards. The line is thin and will be the subject of many legal battles.
Can a judge use ChatGPT to write a sentence? Absolutely not, and it is ethically very serious. ChatGPT "hallucinates" (invents non-existent facts and legal precedents). There have already been cases of lawyers sanctioned for citing cases invented by AI. Justice requires factual truth, not statistical verisimilitude.
How can I know if I have been evaluated by an algorithm? Under the GDPR and the AI Act, you have the right to know if a decision concerning you was made in an automated way and you have the right to request human intervention. Transparency is a fundamental right.
Conclusion: Justice is Not a Calculation
Efficiency is a corporate value, not a judicial one. The goal of justice is not to process the highest number of people in the shortest time possible, but to guarantee a fair trial to every single individual. AI, with its statistical and utilitarian logic, struggles to understand concepts like "mercy," "mitigating circumstances," or "reasonable doubt." As we integrate these powerful tools into our courts and police stations, we must remember that an algorithm can calculate risk, but only a human conscience can understand guilt and, above all, the possibility of redemption.