Why the Real Risk Is Not Artificial Intelligence, but Artificial Morality
From the Roy Batty Paradox to the Universal Declaration of Human Rights
When Pope Leo XIV addressed artificial intelligence in his encyclical Magnifica Humanitas, he did not focus on the question that dominates most public discussions: whether machines will become intelligent enough to replace humans.
Instead, he focused on something far more important.
Values.
Artificial intelligence can already process information, generate content, make recommendations, and support decision-making. The question is no longer whether AI can think. The question is: whose values does AI apply when it thinks?
This distinction may prove to be one of the defining issues of the twenty-first century.
The Wrong Question
Public debate about AI is often framed around intelligence.
Will AI become smarter than humans?
Will it become conscious?
Will it replace workers?
Will it take over the world?
These questions are fascinating, but they may not be the most urgent.
A highly intelligent system that merely calculates is not necessarily dangerous, a system that decides what is fair, acceptable, desirable, or true is something else entirely.
The moment an AI system prioritizes one outcome over another, it is no longer applying logic alone. It is applying values.
And values always come from somewhere.
Why Humans Are Different
Human beings do not merely process information.
They interpret reality.
Every person develops a unique structure of values through a combination of experiences, culture, relationships, education, memories, failures, successes, and personal reflection.
Two people can observe the same event and arrive at completely different conclusions.
This diversity is often perceived as a weakness.
In reality, it may be one of humanity’s greatest strengths.
For years, I have been fascinated by theories of group selection and multi-level evolution. While classical evolutionary theory often focuses on the survival of genes or individuals, some approaches suggest that groups themselves may become units of adaptation.
The evolutionary argument proposed here should not be understood as a strict defense of classical group selection, a concept that remains debated among evolutionary biologists. Rather, it adopts a broader, multilevel perspective in which diversity, cooperation, and altruistic behavior may contribute to the adaptive success of collectives, whether biological, cultural, or institutional.
If this perspective contains even a portion of the truth, diversity is not a flaw in the system.
It is the system.
Groups survive because they contain different types of individuals:
- explorers and conservatives;
- innovators and protectors;
- altruists and competitors;
- risk-takers and risk-avoiders;
- visionaries and pragmatists.
A perfectly homogeneous group may be efficient under stable conditions.
A diverse group is resilient when conditions change.
Humanity’s success as a species may have emerged not despite diversity, but because of it.
The Survival Function of Altruism
This perspective also offers an explanation for something that has puzzled evolutionary thinkers for decades.
Why do altruistic behaviors survive?
Why do individuals sometimes sacrifice their own interests for the benefit of others?
From a purely individual perspective, such behaviors appear maladaptive.
Yet they persist.
One possible explanation is that altruistic traits contribute to the fitness of the collective.
Even if a specific behavior appears costly to an individual, it may strengthen the group that carries the underlying genetic or cultural traits.
Trust, cooperation, empathy, sacrifice, and solidarity may therefore function as evolutionary technologies that improve collective adaptability.
In this view, morality itself becomes part of humanity’s survival strategy.
The individual may occasionally lose.
The collective gains.
And over long periods of time, collectives capable of generating trust and cooperation may outperform those that cannot.
The Roy Batty Paradox Revisited
This is where the Roy Batty Paradox enters the discussion.
The paradox suggests that the fundamental limitation of artificial intelligence is not computational power, it is the absence of lived experience.
Current AI systems derive their capabilities from convergence.
They aggregate information from billions of examples into a single statistical model, and their strength comes from identifying common patterns.
Humans operate differently.
Every human being represents a unique trajectory through reality.
Experience generates interpretation, interpretation generates values, values generate individuality, and individuality generates diversity.
And diversity generates adaptability.
This creates a fundamental dilemma:
If an AI were to acquire genuine moral agency, it would likely require
- autonomous experience;
- persistent memory;
- personal consequences;
- self-directed learning;
- independent value formation.
But these same characteristics would inevitably produce individuality.
Two such systems would eventually diverge from one another, just as two humans raised in different environments diverge.
The closer AI comes to authentic moral agency, the less scalable it becomes.
It ceases to be a product and starts becoming a person.
The Real Risk: Centralized Morality
Ironically, this means that today’s AI systems face the opposite problem.
They are not too independent.
They are not independent enough.
Because they do not possess their own value systems, they rely on externally supplied values.
These values may come from:
- corporations;
- governments;
- regulators;
- developers;
- training datasets;
- political environments;
- economic incentives.
This creates the most significant risk associated with contemporary AI.
The danger is not that machines develop their own morality, the danger is that machines become the perfect vehicle for someone else’s morality.
Throughout history, power has always sought mechanisms capable of scaling its worldview.
Empires used law, religions used doctrine, states used education, mass media used communication, but artificial intelligence may become the most powerful value-distribution system ever created.
Because, unlike previous technologies, it can participate directly in millions of decisions every day.
The concern expressed by Pope Leo XIV points precisely in this direction.
The issue is not intelligence: the issue is authority.
Who decides what values are embedded into the system?
Who decides what constitutes fairness, truth, acceptable risk, dignity, or justice?
And who verifies that those decisions genuinely serve humanity rather than the interests of those who possess the greatest concentration of wealth, influence, data, or computational resources?
The Constitutional Alternative
Many people fear the idea of collective governance for AI.
They worry that international organizations or governments could impose ideological frameworks.
This concern is legitimate, yet there is an equally serious risk:
If AI is not governed collectively, it will inevitably be governed privately.
Private governance is not neutral; every organization is influenced by incentives, interests, power structures, and economic objectives.
The challenge is therefore not to eliminate values; it is to determine which values deserve legitimacy.
Perhaps the closest historical precedent is the Universal Declaration of Human Rights.
After the devastation of the Second World War, humanity attempted something extraordinary.
Not the creation of a universal culture, but the creation of a minimum common ethical framework.
A constitutional foundation.
The declaration was imperfect, and it remains debated and unevenly applied, yet it represented a collective attempt to identify principles sufficiently universal to transcend nations, ideologies, religions, and political systems.
The same principle could guide the future development of AI.
Instead of embedding the values of a corporation, a nation, or a political movement, AI systems could be aligned around broadly negotiated principles such as:
- human dignity;
- human autonomy;
- transparency;
- accountability;
- non-discrimination;
- protection of life;
- freedom of conscience;
- respect for cultural diversity.
Not a complete moral doctrine… again, a constitutional framework.
The Limits of Cultural Relativism
At first glance, this proposal appears to contradict cultural relativism:
If values are context-dependent, how can humanity establish a common set of principles?
The answer may lie in distinguishing principles from applications.
Different cultures often disagree on implementation, they frequently disagree on priorities, they sometimes disagree on interpretations…
Yet they often converge on fundamental aspirations:
- reducing unnecessary suffering;
- protecting life;
- preserving dignity;
- fostering cooperation;
- seeking justice.
The challenge is not to create a universal culture; the challenge is to establish a minimum common denominator that protects humanity from the concentration of moral authority in the hands of a few actors.
The Constitutional AI Paradox
This leads to a final paradox:
A truly autonomous AI would eventually develop its own values through experience.
But such a system would become individualized and lose much of the scalability that makes AI economically valuable.
A scalable AI, on the other hand, cannot generate its own values.
It must inherit them.
The future of artificial intelligence, therefore, seems to split into two paths:
- The first path produces artificial persons.
- The second produces artificial institutions.
The greatest danger may not come from the first; it may come from the second.
Because the question is not whether AI will become intelligent enough to think like us, the question is who will decide what it should think about us.
And whose values will it apply when it does?
References and Further Reading
Artificial Intelligence, Ethics and Human Dignity
- Vatican.va. (2026). Encyclical Letter of His Holiness Leo XIV Magnifica Humanitas (15 May 2026). [online] Available at: https://www.vatican.va/content/leo-xiv/en/encyclicals/documents/20260515-magnifica-humanitas.html.
- Vatican.va. (2024). Participation of the Holy Father Francis at the G7 in Borgo Egnazia (Puglia) (14 June 2024) | Francis. [online] Available at: https://www.vatican.va/content/francesco/en/speeches/2024/june/documents/20240614-g7-intelligenza-artificiale.html.
- Rome Call for AI Ethics. (n.d.). Available at: https://www.vatican.va/roman_curia/pontifical_academies/acdlife/documents/rc_pont-acd_life_doc_20202228_rome-call-for-ai-ethics_en.pdf.
- United Nations (1948). Universal Declaration of Human Rights. [online] United Nations. Available at: https://www.un.org/en/about-us/universal-declaration-of-human-rights.
Evolution, Diversity and Collective Adaptation
- Charles Darwin, On the Origin of Species (1859).
- Hamilton, W.D. (1964). The genetical evolution of social behaviour. II. Journal of Theoretical Biology, [online] 7(1), pp.17–52. doi:https://doi.org/10.1016/0022-5193(64)90039-6.
- Trivers, R.L. (1971). The Evolution of Reciprocal Altruism. The Quarterly Review of Biology, 46(1), pp.35–57. doi:https://doi.org/10.1086/406755.
- Richard Dawkins, The Selfish Gene (1976).
- Stanford, P.K., Sober, E. and Wilson, D.S. (2001). Unto Others: The Evolution and Psychology of Unselfish Behavior. The Journal of Philosophy, 98(1), p.43. doi:https://doi.org/10.2307/2678419.
- E. O. Wilson, The Social Conquest of Earth (2012).
Culture, Values and Human Diversity
- Clifford Geertz, The Interpretation of Cultures (1973).
- Peter Berger & Thomas Luckmann, The Social Construction of Reality (1966).
- Karl Popper, The Open Society and Its Enemies (1945).
Artificial Intelligence and Governance
The Roy Batty Paradox
The Roy Batty Paradox is a conceptual framework developed by, well… me. It argues that the principal limitation of artificial intelligence is not computational capacity but the absence of direct experience and subjective value formation. The paradox further suggests that any AI capable of autonomously building its own value system through lived experience would necessarily become individualized, thereby reducing the scalability that currently constitutes the primary economic and operational advantage of AI systems.