The Moral Question in Automated Decisions: Who Judges the Algorithm? (Between Responsibility and "Moral Agency")
We have entered an era where machines decide who to hire, who to treat, and who to condemn. But when the algorithm errs, who pays? In this article, we explore t
Imagine a scenario, unfortunately no longer hypothetical. An autonomous vehicle must choose between hitting a careless pedestrian or crashing into a wall, killing the passenger. A recruiting software systematically discards women for managerial roles based on historical data. A "credit scoring" system denies a vital mortgage to a family without providing an intelligible explanation.
In all these cases, there is a decision. There is harm. But is there a culprit? We have entered the era of ADMS (Automated Decision-Making Systems). We have delegated to machines the ability to judge, calculate risks, and allocate resources. But while we have transferred decision-making authority, we have not yet figured out how to transfer moral responsibility.
The algorithm has no conscience, feels no remorse, cannot "pay" for its mistakes. And if the machine cannot be responsible, who is? The programmer who wrote the code five years earlier? The company that uses it? Or the user who blindly trusted the result?
In this article, we will explore the concept of Accountability in the age of AI, analyzing "Responsibility Gaps," ethical dilemmas (like the applied Trolley Problem), and emerging regulatory solutions. Because a world governed by algorithms without judges is a world without justice.
1. The Responsibility Gap: The Problem of "Many Hands"
The heart of the moral issue lies in what philosophers call the Responsibility Gap. When a decision is made by a complex neural network (Black Box), the causal link between human action and the final outcome disintegrates.
Distributed Responsibility
As highlighted in a foundational study on ScienceDirect (sciencedirect.com), ethics in ADMS is not an isolated problem, but a systemic one. Cecez-Kecmanovic (2025) argues that responsibility is "distributed." An algorithm is the product of:
- Developers who write the code.
- Data Scientists who select the training data.
- Deployers (companies) who implement the system in a specific context.
- End users who interpret (or misunderstand) the output.
- Regulators who set the standards.
In this chain, it is easy for each to say: "It's not my fault, the system acted unexpectedly." This phenomenon, known as the "Problem of Many Hands," risks leaving victims with no one to hold accountable. If everyone is partially responsible, no one truly is.
The Moral Machine Dilemma
The document published in WJARR (wjarr.com) takes this concept to the extreme with the "Moral Machine Problem." In critical sectors like healthcare, finance, and transportation, AI finds itself having to make choices that involve moral values (e.g., who to treat first in an overcrowded emergency room?). The responsibility gap here is evident: if a doctor makes a mistake, they are liable for malpractice. If an AI makes a mistake because its training data did not contain enough rare cases, is it malpractice? Is it a product defect? Or is it a "statistical misfortune"? Closing this gap requires not only new laws but a new ontology of responsibility.
To delve deeper into how these opaque decision-making mechanisms influence society, we refer you to our article on Algorithmic Bias and Justice: Solutions and Risks.
2. Bias and Discrimination: When is the Algorithm "Bad"?
If the algorithm has no intentions, can it be "immoral"? The answer is yes, if we consider the effects and not the intentions.
Bias as a Moral Violation
According to the analysis available on PhilArchive (philarchive.org), automated decision-making systems can perpetuate historical injustices more effectively than any human. If a hiring algorithm is trained on ten years of CVs from a male-dominated company, it will learn that "being a man" is a predictor of success. Ethics here splits between two approaches:
- Utilitarian: The algorithm is good if it maximizes global efficiency (even if it sacrifices some individuals).
- Deontological: The algorithm is unacceptable if it violates fundamental rights, regardless of efficiency. Democratic society tends towards the deontological approach, but the market pushes towards the utilitarian one.
The Legal Perspective and the GDPR
MediaLaws (medialaws.eu) reminds us that Europe has tried to put a brake on this with Article 22 of the GDPR, which establishes the right not to be subject to decisions based solely on automated processing. However, proving algorithmic discrimination is extremely difficult for the victim (probatio diabolica). For this reason, there is discussion of reversing the burden of proof: it should be up to the company to demonstrate that its algorithm did not discriminate, and not up to the citizen to prove the opposite.
This issue deeply touches civil rights. For an analysis of current protections, read our focus on AI and Protection of Workers' Digital Rights.
3. The Human Factor: Anthropocentrism and "Automation Bias"
One of the proposed solutions to the moral problem is to always maintain a "Human-in-the-Loop." But are we sure that's enough?
AI's Influence on Human Agency
A fascinating study published in Nature Scientific Reports (nature.com) demonstrated that AI behavior influences the perception of human responsibility. In the experiment (a variant of the Trolley Problem), when participants were assisted by an AI that suggested a utilitarian action (sacrifice one to save many), they felt less responsible for the final choice. The AI acts as a "moral scapegoat." This is extremely dangerous: if the judge or doctor blindly trusts the machine's suggestion (Automation Bias), the human presence becomes an empty formality, a fig leaf to legitimize automatic decisions.
The Judge as the Last Bastion
The journal Questione Giustizia (questionegiustizia.it) makes a firm point: mandatory anthropocentrism. In the judicial field, total delegation to AI is unconstitutional. The judge must be morally and legally responsible for the verdict. AI can be a support (auxiliary predictive justice), but it can never replace the discretionary evaluation of the human being, the only one capable of understanding context, fairness, and mercy – concepts that do not exist in binary code.
The psychological interaction between man and machine is complex. Explore how AI influences our mind in our article on AI and Psychology of the Mind: Diagnosis and Algorithms.
4. Regulation and Governance: Who Decides What is Right?
If the algorithm must follow moral rules, who writes these rules? Silicon Valley? Brussels? Beijing?
The Geopolitics of Ethics
Vox Sinattica (vox.sinattica.com) highlights the clash of visions:
- USA: Market-driven approach, focus on innovation, ethics as voluntary "best practice."
- EU: Rights-driven approach (AI Act), focus on protecting fundamental rights, ethics codified into law.
- China: State-driven approach, focus on social order and stability. In a globalized world, an algorithm developed in California (with American values) can decide the fate of a worker in Italy or a dissident in Asia. International governance is the challenge of the decade: cross-cutting ethical standards are needed to avoid "ethical dumping."
Audit and Transparency
From a practical standpoint, the Dutch data protection authority (Dutch DPA), cited by Stibbe (stibbe.com), suggests concrete measures to move forward responsibly:
- Accountability Records: Document every phase of AI development.
- Audit Logs: Record why the AI made a certain decision.
- Generative AI Policy: Clear rules on which data can be used and how risks of hallucination or bias are mitigated.
5. Solutions: XAI and the Lifecycle of Responsibility
We cannot stop automation, but we can make it responsible. How?
The Lifecycle of Responsibility
Fonzi.ai (fonzi.ai) proposes mapping responsibility along the entire product lifecycle:
- Design: Developers must incorporate ethical principles into the code (Ethics by Design).
- Deployment: Companies must monitor AI in the real world (it's not enough that it works in the lab).
- Redress: Clear mechanisms must exist to contest the algorithmic decision.
Explainable AI (XAI) as a Civil Right
EICTA IITK (eicta.iitk.ac.in) identifies transparency as the keystone. This is where Explainable AI (XAI) comes into play. If a system is an impenetrable "Black Box," it cannot be ethical. We must move to "Glass Box" systems or develop interfaces that explain: "The mortgage was denied because the debt-to-income ratio is too high, not because of your ZIP code." Without explainability, there is no accountability. And without accountability, there is no trust.
FAQ: Frequently Asked Questions on Ethics and Algorithms
1. Who is responsible if an AI commits a crime? Currently, criminal liability is personal and cannot be attributed to a machine. The human who used the AI is liable (if there is intent or gross negligence) or the manufacturer (if there is a manufacturing defect). In the future, there is discussion about granting an "electronic personality" to robots for insurance purposes, but it is a controversial topic.
2. What is the "Trolley Problem" applied to AI? It is an ethical thought experiment: a driverless vehicle with no brakes must choose whether to hit five pedestrians or swerve and kill the passenger. How is this choice programmed? Utilitarianism (save the greatest number of lives) or customer protection (save the passenger)? There is no universally correct answer.
3. Does Article 22 of the GDPR prohibit automated decisions? It does not prohibit them absolutely, but it gives the citizen the right not to be subject to decisions based solely on automated processes that have significant legal effects (e.g., dismissal, credit denial), unless there is explicit consent or contractual necessity. In any case, it guarantees the right to human intervention and to contest the decision.
4. Can AI be less racist than humans? In theory, yes. If correctly programmed and cleansed of biases from historical data, AI could apply rules more consistently and impartially than a tired or prejudiced human judge. But practice shows that "cleansing" data is extremely difficult because bias is often structural in society.
5. What are Ethical Review Boards? They are ethical committees internal to companies or external (governmental) that evaluate the impact of an algorithm before it is released on the market. They function like bioethics committees for drugs: they analyze risks, benefits, and potential harm to human rights.
Conclusions: From Blind Trust to "Trustworthiness"
The moral question in automated decisions will not be solved with a software update. It requires an update of our legal and philosophical system. We cannot treat the algorithm as an infallible oracle, nor as a scapegoat for our social failures.
The algorithm is a mirror: it reflects our values, our biases, and our priorities. If we don't like the image we see reflected – an efficient but unjust world – we must not break the mirror (the AI), but change who stands in front of it (ourselves and our rules). Who judges the algorithm? In the end, it must be us. With the tools of law, technology (XAI), and, above all, human critical consciousness, the only "black box" capable of doubting itself.
Bibliographic References and Sources
To ensure the accuracy and authority of this analysis, the article has drawn from the following primary sources: