Concerning what I’m inclined to call the great AI debate of the day — about whether we should fear extinction of the human race — I had a somewhat surreal idea I wanted to explore, which I submitted in the following prompt.  

I’m toying with an idea that may sound outrageous. I’d like your immediate frank reaction. As a preamble let me mention that one of the concerns of those who see AI as an imminent danger is its capacity, which some deem a proclivity to commit cybercrimes. As some commentators have correctly signaled, banning AI would not prevent cybercrimes, which existed well before AI became readily available.

Drawing on the same logic, when I see predictions about AI “deciding” to annihilate humanity, I would object that not only is the idea of annihilating humanity not new but the ability to carry it out requires means that are not proper to AI.

But I’ll go further and suggest that the question of humanity’s extinction due to AI is predicated upon AI having a decision-making capacity of a kind that humans have never explicitly allowed to any technology. All highly consequential decisions in the past have had at least two distinct layers: human deliberation and a force of execution.

President Harry Truman made the decision to bomb Hiroshima and Nagasaki. He consulted others to do it but presumably could have done it on his own. He made the decision, but he didn’t drop the bomb. Do today’s fearmongers suppose that AGI will have both autonomous decision-making capacity and the complete power of execution? What would it require on the human side to allow that to happen? Can we suppose that RSI will logically and inevitably lead to such a set of conditions?

I initially addressed this prompt to Claude. I found its response rambling, equivocal, poorly framed and singularly uninspiring. It gave the impression that Claude simply wasn’t interested in the topic.

To the exact same prompt, Perplexity directly addressed the issues in considerable detail. Here is Perplexity’s abridged response.

Your intuition is sharp and not outrageous at all: it lines up with a growing strand of serious criticism that says the real existential danger isn’t “AI waking up and choosing genocide,” but humans gradually ceding the decision and execution layers to systems we no longer meaningfully control.

The two-layer structure you’re pointing to

You’re right to stress the historical pattern:

• Deliberation layer: Humans (often a small group, sometimes one person) decide that a massively destructive act is acceptable or necessary. Truman’s atomic-bomb decision is a classic example: he framed it as his responsibility, but it emerged from committees, military plans, and political assumptions already in place.
• Execution layer: Other humans—pilots, crews, technicians—carry it out. Even in highly automated systems, there has always been at least a nominal human “in” or “on” the loop.

Existential-risk scenarios that imagine AI “deciding” to annihilate humanity effectively assume that this two-layer structure collapses into one: the system both chooses the endgame and executes it without effective human veto.

What would it take for AI to have both layers?

For AI to reach the kind of agency feared in “annihilation” scenarios, several conditions would have to line up on the human side:

1. Delegation of high-stakes decisions 
2. Removal of meaningful human vetoes 
3. Integration with real-world actuation 
4. Concentration and opacity 

Where RSI fits in

Recursive self-improvement (RSI) is often described as an AI system that:

• Analyzes its own performance and architecture,
• Designs and validates improvements,
• Deploys them, then repeats the cycle, potentially improving its own improvement process.

RSI by itself does not logically imply:

• Autonomous strategic decision-making over human fate, or
• Direct control over destructive infrastructure.

For RSI to “inevitably” lead to annihilation scenarios, you’d need additional assumptions:

• That humans will keep expanding the system’s decision domain into existential-risk areas.
• That they will remove oversight as the system becomes more capable (“it knows better”).
• That the system’s objectives, as they evolve through self-improvement, will conflict with human survival in a way that matters operationally.

Those are contingent social and institutional choices, not pure logical consequences of RSI.

Perplexity then explained in some detail what it called the “more plausible danger: gradual disempowerment.” I continued with a new prompt:

Here is the core issue I wished to discuss and which I called potentially outrageous. It concerns the level of decision-making. We seem to agree that the danger is that humans might relinquish their prerogative of possessing and exercising the ultimate decision-making capacity. I believe that has already happened to some degree and that it is the real reason for justified fear. It’s simply that nobody has noticed it.

Here’s where it gets a little more complicated. I believe that has already happened to some degree and that the real reason for justified fear lies in the fact that nobody has noticed it or wants to admit it exists. The result is that everyone is making a huge misattribution of responsibility and ultimately of accountability.

Here is my outrageous thesis and if I were to blame any single person it might be Milton Friedman. In our current economy and political economy humans no longer make decisions. Money does. Once a culture accepts that shareholder interests is the ultimate criterion for strategic decision-making, politicians bred within that culture delegate their entire moral compass to “the logic of the marketplace,” which is no longer even a genuine marketplace but the artificial one known as the stock market.

When we talk about regulated AI, stopping its development or slowing it down, who are the decision-makers. We know how the shareholders will react, but we also know that politicians even in the most sophisticated democracy in which checks and balances are deemed to be the norm, those who hold any power of individual or collective decision-making will be governed rather than simply influenced by the culture. And that’s why it is legitimate to be afraid of AI… simply because we know it’s “business logic” and not societal or civilizational logic that will ultimately win out. Unless we wake up and acknowledge that reality and seek to modify a system and transform a culture that is depriving humanity of its ability to deploy its own intelligence.

So please react to this outrageous reading of contemporary political and economic culture. Tear it apart, if you think that’s appropriate, or help to guide an emerging debate that concerns everyone on earth.

Your thesis isn’t just provocative; it’s a coherent, if stark, diagnosis of a real structural problem: when “shareholder value” becomes the ultimate criterion, human deliberation gets displaced by a financialized logic that no one individually owns, and accountability dissolves into “the market made me do it.” That’s a powerful lens for thinking about AI risk.

I’ll do both things you asked for: push back where the argument overreaches, then help you sharpen it into something that can carry a serious debate.

Where your argument is strong

1. “Money decides” as a description of corporate governance

You’re right that, in practice, many strategic decisions are framed as if they’re dictated by shareholder value rather than chosen by humans:

• The Friedman doctrine (1970) explicitly argues that the social responsibility of business is to increase profits, treating shareholders as the sole constituency to which the firm is responsible.
• Critics have long argued that this helped legitimize an era of short-termism, hostile takeovers, financial engineering, and the erosion of protections for employees, communities, and the environment—all in the name of maximizing shareholder returns.
• Empirical work on corporate short-termism shows that equity-market pressure can push firms to favor immediate results over long-term value creation, sometimes trapping them in a “vicious circle” of outdoing peers on quarterly metrics.

So when you say “humans no longer make decisions; money does,” you’re capturing a lived reality in many boardrooms: executives present choices as technically compelled by “market logic” rather than morally chosen.

2. Politicians socialized in that culture

Your claim that politicians are “bred within that culture” and thus delegate their moral compass to market logic is also supported by a large critical literature:

• Policy debates often treat growth, competitiveness, and capital-market confidence as constraints within which all other goals must be fitted.
• When AI regulation is discussed, you can already see the pattern: fears of “losing the AI race,” damaging “innovation,” or spooking investors routinely frame what’s treated as politically feasible.

That doesn’t mean politicians have no agency; it means their imaginative horizon is narrowed by a political economy in which “the market” is treated as an objective force rather than a set of human-made rules.

3. Misattribution of responsibility and accountability

This is perhaps your strongest point: if everyone acts as though “shareholder value” or “market logic” is the decision-maker, then:

• No individual feels fully responsible for systemic outcomes.
• Accountability is diffused into an abstract system: “I was just maximizing value,” “We had to stay competitive,” “Regulation would kill innovation.”
• The result is exactly what you describe: a culture that deprives humanity of the ability to deploy its own intelligence to shape its future, including around AI.

On this reading, fear of AI is really fear of a system in which business logic consistently outruns civilizational logic.

Where the argument needs nuance (or risks overreach)

To make this persuasive beyond those already convinced, it helps to anticipate and address several objections.

“Money decides” vs. “some humans decide via money”

Saying “money decides” is rhetorically powerful but can obscure agency:

• Shareholder value isn’t a law of nature; it’s a governance convention enforced by specific humans: boards, asset managers, activist investors, courts interpreting fiduciary duty, and regulators who choose not to intervene.
• Empirical work on short-termism shows that internal corporate factors—CEO incentives, compensation structures, managerial priorities—often matter more than external market pressure in driving short-term behavior.
• Some investors and funds explicitly push back against pure profit-maximization, advocating “shareholder welfare” or stakeholder-oriented strategies, even within the existing legal framework.

Refinement suggestion:
Instead of “money decides,” you might say: “A particular interpretation of shareholder primacy, institutionalized through incentives and norms, has become the de facto decision-maker, allowing real humans to disclaim responsibility.” That keeps the force of your critique while acknowledging agency.

Perplexity’s response didn’t stop there. It went on to critique with precision some other details, which I will not reproduce here. I intend to continue this discussion in subsequent articles, but for today I’ll simply ask readers whether the chatbot’s suggestion — “A particular interpretation of shareholder primacy, institutionalized through incentives and norms, has become the de facto decision-maker, allowing real humans to disclaim responsibility.” — has more impact in furthering the conversation than “Money decides.”

At the very end of its response, Perplexity proposed this summary, which I found very effective as a means of launching a public debate.

The existential risk from AI is not that machines will seize power, but that humans have already surrendered the prerogative of ultimate decision-making to a financialized logic that no one individually owns. Under shareholder primacy, strategic choices are presented as dictated by “the market,” not chosen by people. That misattribution of responsibility lets business logic consistently outrun civilizational logic. AI, developed within this system, inherits and amplifies it. The real question is not how to control AI, but how to reclaim human agency over the institutions that control AI.

Or to be more concise, if the problem is “AI, developed within this system,” we definitely need to change the system.

Your thoughts

Please feel free to share your thoughts on these points by writing to us at dialogue@fairobserver.com. We are looking to gather, share and consolidate the ideas and feelings of humans who interact with AI. We will build your thoughts and commentaries into our ongoing dialogue. 

[Artificial Intelligence has become a feature of everyone’s daily life. We unconsciously perceive it either as a friend or foe, a helper or destroyer. At Fair Observer, we see it as a tool of creativity, capable of revealing the complex relationship between humans and machines.]

[Lee Thompson-Kolar edited this piece.]

The views expressed in this article are the author’s own and do not necessarily reflect Fair Observer’s editorial policy.