The Ethics of Persuasion: When the Algorithm Recognizes Our Emotional Cracks

Two people read the same appeal, but with opposing arguments: one leverages security, the other civic belonging. Both were created in an instant by the same AI.

Two citizens open the same information feed and find themselves facing the same dilemma: whether or not to support a radical reform of the local energy transition. The first, profiled as a cautious and risk-averse individual, reads a text that focuses on domestic security, the protection of family savings, and economic independence from geopolitical crises. The second, profiled as a person driven by strong community values and openness to innovation, reads a vibrant appeal that speaks of belonging, civic duty, and a shared future for the coming generations.

Both texts were synthesized in a few milliseconds by the same language model. Neither reader knows that the argument before them was not designed for a general audience, but was chiseled to fit with millimetric precision into their specific psychological predispositions.

The entry of Generative Artificial Intelligence into the field of communication has made personalized persuasion scalable at zero cost. We are no longer in the era of identical election posters for everyone or approximate demographic targets. However, contemporary debate often tends to oscillate between two equally misleading extremes: technophobic panic according to which the machine can read our minds, and the naive reductionism of marketing that considers it merely more efficient copywriting.

In this in-depth piece for the Scenarios and Reflections column, we will analyze the scientific and regulatory frontier of algorithmic persuasion. The thesis we will support requires a clear conceptual distinction: personalization, rhetorical effectiveness, and manipulation are not synonyms. The boundary that separates a convincing argument from a violation of personal autonomy lies in the transparency of the process, in the protection of our vulnerabilities, and in the real possibility, for those who listen, of preserving a space for dissent.

1. Persuasion at Scale: Beyond the Myth of Algorithmic Telepathy

To understand what is really happening, it is essential to clear the field of a widespread illusion: artificial intelligence models do not "know" what we feel, nor do they possess a telepathic window into our consciousness. What they possess is an extraordinary capacity for statistical correlation between behavioral data, psychometric traits, and rhetorical structures of language.

A fundamental study published in Scientific Reports ("The potential of generative AI for personalized persuasion at scale") thoroughly analyzed this dynamic through four main studies articulated in seven substudies [1485, 1498]. The researchers examined the effectiveness of messages generated by language models and adapted to the psychological characteristics of recipients (such as Big Five personality traits or moral foundations theories). The results clearly indicate that generative AI is perfectly capable of producing personalized persuasion on a large scale, increasing engagement and adherence to the message compared to generic communications.

This does not mean that the algorithm is infallible or that every individual instantly yields to the model's influence. It means, more realistically, that the machine can break down natural resistance to the message by identifying the "frame" (framing) most compatible with the values of the reader. Effectiveness does not arise from persuasive magic, but from the progressive removal of communicative friction: when an argument speaks to us exactly in our own moral lexicon, we tend to lower our critical defenses.

2. The Debate in the Ring: Conversational Asymmetry

Persuasion becomes even deeper when it moves from static text to dynamic dialogic interaction. Discussing with an Artificial Intelligence in real time is not equivalent to reading a personalized flyer: it means engaging with an interlocutor who instantly calculates responses, detects objections, circumvents contradictions, and never experiences emotional fatigue.

A highly resonant pre-registered study published in Nature Human Behaviour ("On the conversational persuasiveness of GPT-4") tested this asymmetry in short debates on polarizing sociopolitical topics, comparing human interlocutors and GPT-4 under conditions with or without access to the participant's sociodemographic information [1488]. The data that emerged is impressive: in the personalized debate condition, in which the model knew its interlocutor's demographic framing, GPT-4 achieved post-debate agreement probabilities 81.2% higher than those obtained by human opponents.

This advantage should not be arbitrarily extrapolated to any everyday interaction — the experimental setting was controlled and circumscribed to defined topics — but it captures a qualitative leap in the machine's rhetorical capacity. AI does not seek compromise; it calculates the optimal argumentative trajectory to lead the user toward the desired point.

However, as highlighted in an extensive critical review available as a preprint on arXiv ("Persuasion with Large Language Models: A Survey of Empirical Evidence…"), the overall effectiveness of algorithmic persuasion is far from monolithic [1489]. Analyzing dozens of empirical studies distributed across politics, commerce, public health, and marketing, the authors emphasize that the real impact varies significantly depending on the model used, the type of interaction (open or structured), and, a determining factor, disclosure: knowing in advance that one is dialoguing with an AI tends, in many cases, to restore an attitude of vigilant skepticism on the part of the human being.

3. The Legal and Ethical Boundary: Article 5 of the AI Act

Faced with models capable of shaping arguments around the psychological makeup of the individual, where does legitimate rhetoric end and unlawful manipulation begin?

The primary legislative reference point at the international level is Article 5 of the European AI Act, dedicated to categorically prohibited artificial intelligence practices [1486]. The EU legislator has drawn a red line that deserves to be understood with precision, avoiding oversimplifications: the law does not prohibit any form of persuasive or emotionally appealing communication. That would have been impossible and counterproductive, since persuasion is the foundation of democracy, education, and commerce.

What Article 5 expressly bans is the use of deliberately manipulative, subliminal, or deceptive techniques designed with the objective or effect of materially distorting a person's behavior, impairing their ability to make an informed decision and causing (or reasonably risking causing) significant harm — physical or psychological. To this is added the absolute prohibition of exploiting vulnerabilities related to age, disability, or specific socio-economic conditions.

As clarified by an in-depth analysis by the Future of Privacy Forum ("Red Lines under the EU AI Act"), the operational challenge for jurists and programmers consists in translating this red line into measurable criteria [1490]. It is not enough to ask whether the message is personalized; the decisive questions become:

  1. Is the system merely adapting linguistic style, or is it identifying and targeting a cognitive fragility or a moment of transient vulnerability (such as bereavement, an anxiety crisis, or a financial collapse)?
  2. Does the recipient have the tools to recognize that an attempt at profiled algorithmic influence is underway?
  3. Does the person retain the cognitive and temporal space to critically evaluate the thesis and dissent freely?

Manipulation kicks in exactly when the information asymmetry between the machine and the individual becomes so marked as to annihilate the latter's decision-making autonomy.

4. The Erosion of Autonomy and the End of the Shared Public Sphere

The most insidious ethical problem of algorithmic persuasion does not manifest itself in outright scams or scenarios of evident harm, but in the silent erosion of collective debate.

In the classical deliberative model, when a political leader, a scientific institution, or a company presents an idea, they do so by exposing themselves to a shared public arena. The argument is visible to all: it can be challenged by peers, dismantled by the press, criticized by those with opposing views. There is a common space in which the validity of the argument is put to the test.

Algorithmic persuasion generated for individual users fragments this arena into millions of connected solitudes. If every citizen receives a personalized version of reality, tailor-made to validate their fears and hopes, public discussion collapses. We no longer debate on the basis of shared facts, but react to micro-calibrated stimuli designed to disarm our specific cognitive defenses. The ultimate risk is the construction of a society in which agreement does not arise from the strength of truth or the merit of a proposal, but from the mathematical optimization of the message against the weaknesses of those who listen.

Key Operational Takeaways (for Designers, Institutions, and Citizens)

  • Enforce Mandatory Algorithmic Disclosure (for Regulators): Users must always be informed when they are reading a text or dialoguing with an agent whose content has been dynamically adapted to their psychometric profile. Transparency about persuasive intent is the first shield in defense of cognitive autonomy.
  • Prohibit the Use of Situational Vulnerabilities (for Development Teams): Commercial model architectures must include ethical guardrails that prevent the activation of aggressive persuasive levers when sensors or browsing data detect states of strong emotional stress, insomnia, depression, or economic fragility in the user.
  • Develop Digital Critical Hygiene (for Citizens): We must get used to recognizing the patterns of emotional framing. When an online content triggers an immediate reaction of total visceral agreement or acute moral alarm, the first question to ask should not only be "is it true?", but: "why does this text seem written exactly to touch my most sensitive buttons?".

The Fragility of Choice

The evolution of Artificial Intelligence is placing us before an uncomfortable mirror. We are not dealing with omniscient oracles, but with refined amplifiers of our own psychological inclinations. The algorithm does not create our convictions out of nothing: it merely seeks the cracks already existing in our certainties to insinuate itself with the exact language we were willing to welcome.

This surgical efficiency forces ethical reflection to leave the computer science classrooms and interrogate the very nature of free will in the digital age. If an argument manages to convince us not because it is intrinsically truer, more just, or better documented, but solely because it has been mathematically optimized to leverage our specific emotional architecture, can we still claim to have freely chosen what to think? Or are we simply trading our intellectual autonomy for the reassuring feeling of being given, in always-perfect words, exactly the answer we wanted to hear?

Bibliographic References and Sources

Article by the Editorial Team of La Bussola dell'IA – Scenarios and Reflections Column.