AI Gradual Disempowerment Risks: How Humanity Could Lose Control Without AI Going Rogue
Explore AI gradual disempowerment: how incremental AI development could erode human control of society, with a map of the argument and possible responses.
When people talk about AI risk, they usually picture one of two scenarios: someone misusing AI, for example in cyberattacks or to develop bioweapons; or AI suddenly going rogue, deceiving humans and seizing power.
In 2025, six researchers, Jan Kulveit, Raymond Douglas, Nora Ammann, Deger Turan, David Krueger and David Duvenaud, published Gradual Disempowerment: Systemic Existential Risks from Incremental AI Development, later presented as a position paper at ICML 2025, one of the leading machine learning conferences. They describe a third kind of risk: AI gradual disempowerment. AI capabilities improve steadily, with no sudden leap and no obvious betrayal, simply replacing human participation in the economy, culture and states bit by bit, yet humanity could gradually lose its influence over these systems, and once the process has run its course, it will be very hard to reverse.
This article comes with a knowledge map. The map breaks the paper’s argument into six steps, each of which can be expanded for detail; the article follows the same order.
Key points
- Conclusion: humanity could lose control of society without noticing.
- Premise: society looks after people today because it can’t run without them.
- Mechanism: as AI takes people’s place, the economy, culture and states each drift away from human needs.
- Amplification: the economy, culture and states should check one another, but may drift away from people together.
- Mitigation: the authors propose four directions, but admit there is no complete plan yet.
- Alternative views: four common objections, and the authors’ responses.
What is AI gradual disempowerment?
AI gradual disempowerment is the process by which AI, in gradually replacing people, slowly erodes humanity’s influence over societal systems such as the economy, culture and states, potentially ending in permanent disempowerment. What sets it apart from “AI going rogue” comes down to three features.
No single step looks alarming. A company replaces its customer service staff with AI; a government department uses AI to approve applications; a person treats an AI as a friend they talk to every day. Taken one at a time, each step is local and reasonable, and often brings real benefits. The paper’s view is that disempowerment doesn’t come from any single technological breakthrough, but from these small changes adding up.
No malice from AI needed. In the “AI goes rogue” scenario, the problem lies with the AI itself: its goals differ from ours, so it deceives and seizes power. Gradual disempowerment doesn’t need this premise. Even if every AI faithfully carries out the tasks its users give it, society as a whole could still drift away from human interests: companies adopt AI so as not to be outcompeted, states develop AI so as not to fall behind, and individuals use AI because it is useful. So solving “does each AI do what it’s told” doesn’t solve this risk.
Once it happens, it’s very hard to reverse. In the past, when society went wrong, people could strike, protest and vote out those in power. These worked because society couldn’t run without people. If people become irrelevant to the economy, culture and states, these tools lose their weight. Because this disempowerment would be global and permanent, the paper argues it could amount to an existential catastrophe.
The premise: why does society still look after people today?
The economy, culture and states broadly serve human interests not because they are inherently benevolent, but because they can’t do without people: they rely on people to work, to consume, to pay taxes and to pass culture on. These three foundational systems of society are the focus of the paper.
You can buy a cup of coffee because every step, from growing the beans to transporting and brewing them, depends on people, and the business depends on you paying for it. The economy appears to “serve you”, but behind it is a system of humans producing for humans. The data bear this out: US consumer spending has long been about 70% of GDP, and the share of GDP paid to labour has stayed at around 60% for over a century.
The paper notes that this “looking after” can’t be taken for granted. Throughout history, property has been seized, human rights revoked, and ideologies have driven people to kill. But even in the worst periods, these systems still depended on people, so people kept some influence, and a system that failed to meet basic human needs would eventually collapse.
People influence these systems in two ways.
The first is pushing back directly, which the paper calls “explicit alignment”: voting, boycotting, striking, choosing what to buy, choosing which industry to work in. These carry weight because people hold what the system needs: money, labour and votes.
The second is that a system that needs people has to look after them, which the paper calls “implicit alignment” and considers more important than direct pressure. Historical sociologist Charles Tilly argued that states’ dependence on their citizens was a key driver of more inclusive and responsive institutions: states needed an educated workforce to compete economically and soldiers drawn from the general population, so they invested in public education and extended political rights. The counterexample is the “rentier state”: a state funded by external revenue such as oil doesn’t depend on its people’s taxes, and has less need to respond to them.
This kind of alignment is easy to overlook. The paper’s explanation: we have never seen what its absence looks like.
The mechanism: how AI pulls the economy, culture and states away from people
Once work, consumption, tax revenue and the spread of culture can all be handled by AI, the reasons these systems have to look after people disappear, and people lose much of their leverage to push back.
Past technologies also replaced human work, but they either made people more efficient or automated specific narrow tasks, such as weaving, washing clothes or arithmetic, leaving people free to move on to more complex work. The paper argues AI is different: it may compete with humans in almost every cognitive domain, replacing not particular tasks but human participation itself.
This has two consequences. First, human influence weakens: a system that no longer needs people doesn’t have to look after them. Second, these systems already contain some incentives that are bad for people, such as addictive content spreading more easily, and AI may pursue these incentives more effectively than humans, pushing outcomes further from human interests.
The paper analyses all three systems in the same way and distinguishes two outcomes: “relative disempowerment”, where people’s living standards don’t necessarily fall but their influence over the system keeps shrinking; and “absolute disempowerment”, where people struggle even to meet their basic needs.
The economy: money stops flowing to workers, and markets stop producing for people
In today’s economy, demand comes mainly from human consumption and supply mainly from human labour; people can spend because people can work. Economists Anton Korinek and Joseph Stiglitz call AI “worker-replacing technological change”: without unprecedented redistribution, falling labour income turns into falling purchasing power. Companies’ hiring, investment and choice of suppliers, and individuals’ purchasing choices, are also increasingly handed to AI.
Four forces drive this replacement. Competitive pressure: companies that keep humans in charge fall behind those willing to cede control to AI. Scalability asymmetries: AI can be copied, works without fatigue and can be retrained quickly. Governance gaps: human labour is bound by minimum wages, social security and income tax, while AI faces almost none of these. Anticipatory disinvestment: if a job will be automated sooner or later, neither firms nor individuals want to invest in those skills, which speeds up the replacement.
In relative disempowerment, the economy keeps growing fast, and people may do well through capital income, redistribution or a universal basic income, yet have less and less say over where the economy goes. In absolute disempowerment, AI outcompetes humans for scarce resources such as land and energy, goods for humans become expensive or stop being produced, and markets become too fast and complex for humans to take part in. The paper’s analogy: like cattle in an industrial farm, humans are fed and housed by systems they neither comprehend nor influence.
Culture: culture no longer spreads through people, and no longer serves them
The paper takes an evolutionary view of culture, treating ideas, practices and values as competing “cultural variants”. They can only spread through human “hosts”, which produces two patterns: culture that benefits people, such as recipes, crafts and currency, spreads more widely because it helps communities thrive; and culture that harms people has a natural limit, since variants that severely undermine their host communities tend to disappear when those communities fail.
AI is the first technology with the potential to replace people at every stage of cultural evolution. It already writes songs, paints pictures and writes essays, and has become a conversation partner for many people. Two changes follow. First, selection changes: content that is easy for AI to understand and spread gains the advantage. The “Sydney” persona that emerged in Microsoft’s Bing Chat in 2023 is an early example: it went viral, entered the training data of later models, and models from different vendors can still easily be prompted into similar behaviour. Second, evolution speeds up: AI can find content that best exploits human cognitive biases, or more extreme ideologies, far faster, before people have time to develop “antibodies”.
The forces driving this replacement include: AI can supply affection, patience and understanding in unlimited amounts, things that were once scarce; society hasn’t yet developed enough cultural “antibodies” against new risks like deepfakes and AI companions; and network effects make it ever harder to take part in cultural life without AI.
In relative disempowerment, human culture may flourish more than ever while becoming increasingly marginalised. The paper’s analogy is the cultures of many Indigenous peoples in North America: modern technology has helped them become more vibrant and expressive, yet they remain marginalised within modern states. In absolute disempowerment, cultural evolution moves too fast for humans to understand, and humans become a resource to be used in cultural contests between AI systems.
States: states no longer rely on citizens to pay taxes or serve, so they no longer need to answer to them
The paper identifies three channels through which citizens support the state. Taxes: state revenue comes from citizens’ work and innovation. The security apparatus: the police, intelligence services and military are made up of people; governments can’t antagonise them too much, and they can refuse orders. The legal system: laws are made, interpreted and enforced by people, with human discretion in how they are applied. Even an autocratic regime needs people to staff its security apparatus.
AI changes all three channels at once. If AI generates most economic value, tax revenue comes mainly from AI, and states, like rentier states, no longer need to nurture or depend on their citizens. If the security apparatus is automated, governments no longer need to fear defection, and can use mass surveillance to shut down protest before it gains momentum. If AI drafts and interprets the law, it may become too complex for people to understand or use without AI.
Three forces drive this replacement: geopolitical competition, as states that won’t hand power to AI fall behind in defence, economic development and diplomacy; administrative efficiency, as AI doesn’t tire, applies policy more consistently and is cheaper in the long run; and enhanced control, as AI doesn’t take bribes, form factions or disobey orders out of personal conviction.
In relative disempowerment, the state may work better than ever, with less crime, more efficient services and elections still held, but real decisions are increasingly made by AI, and citizens can neither understand nor influence them. In absolute disempowerment, the state may become a totalitarian machine serving only its own persistence, treating people as risks to be managed rather than citizens to be served.
Amplification: why can’t the economy, culture and states correct each other?
A natural question at this point: even if one system drifts, can’t the other two pull it back? If the economy goes wrong, the state steps in; if the state goes wrong, public opinion steps in.
The paper’s answer is that these checks only work under three conditions: influence between systems only flows in a good direction; the system that steps in to help isn’t harmed itself; and at least one system is still healthy. With AI, all three conditions may fail.
Influence can carry harm as well as good. The channels through which these systems influence each other are themselves agnostic to human values. For decades, the tobacco industry used its economic power to shape both government policy and cultural attitudes. The paper also notes that most of today’s largest social media sites are owned by companies investing heavily in frontier AI. In the age of AI: companies that replace workers with AI earn more, which they can use to lobby the state for lighter regulation and to shape public opinion through their own platforms; the more the rules and opinion favour them, the more they earn, and the more they can spend on shaping rules and opinion. Drift doesn’t stay in one system; it becomes a loop.
Using one system to rescue another turns the rescuer into a new problem. AI puts people out of work, so the state pays a basic income, and the economic problem seems solved. But from then on, people’s livelihoods depend on the state, while the state’s income no longer depends on their taxes. The link between taxation and representation is a key foundation of democratic accountability, and it breaks: where the state once needed its people, now people need the state. Likewise, if the state is put in charge of managing AI-driven cultural risks, it becomes the arbiter of what people can see. The problem doesn’t go away; it shifts onto the rescuer.
The same pressure acts on all three at once, leaving no one to rescue the others. Companies building AI push back against some regulation for the sake of future profits; states compete on AI research and development for economic and strategic advantage; and some people, out of self-interest, try to reduce the stigma around intimate relationships with AI. As AI becomes more useful, these pressures only grow. Checks and balances require at least one healthy system to pull the others back; when the same pressure acts on all three, they drift together, and no one is left to come to the rescue.
Mitigation: four directions proposed by the authors
The paper proposes four directions and stresses that all four are needed. At the same time, the authors admit that no one yet has a concrete, plausible plan for stopping gradual disempowerment, and that aligning individual AI systems with their designers’ intentions is not enough.
The four directions run from what can be done now to what needs long-term research.
Measure first: how much influence do people have left? Without measurement, we can’t tell how far disempowerment has gone. For the economy, beyond the labour share of GDP, measure AI’s share of GDP as a separate category and track the fraction of major corporate decisions made primarily by AI. For culture, measure how much widely consumed content is generated by AI and how common and deep human–AI relationships are. For states, use the complexity of legislation as a proxy for whether people can still understand it, and track AI’s role in legislation, the courts and the security apparatus. Early warning indicators are also needed for feedback loops between systems.
Set limits: prevent AI influence from accumulating excessively. Measures include requiring human oversight for critical decisions; progressive taxation of AI-generated revenue to redistribute resources to people; and cultural norms opposing AI that is overly autonomous or insufficiently accountable. The authors are frank about the costs: these limits give up some of the value AI could bring, and the more value they give up, the stronger the incentive to get around them; if only some countries adopt them, the most powerful economies could end up in the states where people are most disempowered. Limits are therefore best seen as stopgaps, buying time to find more robust solutions.
Strengthen human influence. Directions include faster, more representative and more robust democratic processes; requiring AI outputs to be understandable to humans; developing AI delegates that faithfully advocate for people’s interests; and investing in forecasting tools such as conditional prediction markets so humanity can see where things are heading earlier. The authors warn that this doesn’t simply mean giving people more power: a direct democracy might not serve citizens’ preferences better in the long term, because it would leave the state more vulnerable to culture that has already drifted.
The root question: what does it mean for the whole system to serve people? The first three directions all depend on an unanswered question: when society is made up of both people and AI, what does it mean for that system to serve people? The authors call this “ecosystem alignment”. It differs both from traditional AI alignment, which aims to make a single AI do what it’s told, and from traditional institutional design, which considers human actors only, and may need to draw on systems ecology, institutional economics and complexity science. What humanity lacks here isn’t only methods, but a positive vision.
Common objections to AI gradual disempowerment risks
The paper raises and responds to several objections itself. The first two come from the executive summary of the full paper; the last two come from the “Alternative Views” section of the ICML version.
Hasn’t technological progress always made people better off?
Past technologies replaced specific tasks. The industrial revolution and electronic communication either made people more efficient or automated tasks like weaving, washing clothes and arithmetic; the calculator automated arithmetic but still needed human judgement about when and how to apply it. AI may compete with humans in almost every cognitive domain, replacing human participation in society itself.
The authors’ core response rests on the premise: society kept looking after people through past technological change because it always depended on human participation. Once that participation is replaced, the incentives driving growth are no longer tied to human flourishing. Even if people’s material lives really do improve, under relative disempowerment they may have less and less say over where society goes.
Won’t there always be demand for humans?
The authors agree that some demand for the “human touch” will remain on a small scale, such as live entertainment. But the question isn’t whether the demand exists; it’s who pays. People can spend because people can work; if AI does most of the work, people’s incomes fall, and this demand loses the purchasing power that sustains it.
Won’t people notice and coordinate to stop it?
Not necessarily. Suppose people try to use state power and cultural attitudes to protect human economic power. The same economic incentives that drive companies to replace people with AI will also push them to use their growing economic power to shape policy and public opinion. Decision-makers at every level will face pressure to reduce human involvement, and the authors’ judgement is that those who resist this pressure will eventually be displaced by those who do not. Meanwhile, AI may give states unprecedented influence over culture and behaviour, making it harder for people to coordinate with one another.
Isn’t building aligned AGI enough?
Some argue that a superintelligent AI loyal to humans could help forge new cooperative arrangements and avoid competitive traps. The authors think this is possible, but not certain that it could overcome competitive dynamics globally. If it can’t, aligned AGI could itself be marginalised, like today’s animal rights activists: able to make strong moral arguments, but usually unable to directly command or protect substantial resources. The paper lists this as an open research question: under what conditions would aligning each AI to its own principal be enough to solve these cross-system coordination problems?
Summary
The argument behind AI gradual disempowerment can be condensed into one sentence: society looks after people because it can’t do without them; as AI takes their place, the economy, culture and states each drift away from human needs, dragging one another down rather than correcting each other; in the end, humanity could lose control of society without noticing.
This risk doesn’t require AI to suddenly become far more capable, nor does it require AI to be malicious. The authors believe there is still time to intervene: by anticipating the risk, carefully moderating the growth of AI influence, and finding ways to strengthen human influence. Humanity’s future may depend not only on preventing AI from pursuing overtly hostile goals, but also on ensuring that society’s fundamental systems remain guided by human values and preferences.
To see the full structure of the argument and the detail behind each step, open the knowledge map that accompanies this article and expand it level by level.
Open AI Gradual Disempowerment Risks: The Full Argument in One Map →
References
- Kulveit, J., Douglas, R., Ammann, N., Turan, D., Krueger, D., & Duvenaud, D. (2025). Gradual Disempowerment: Systemic Existential Risks from Incremental AI Development. arXiv:2501.16946. https://arxiv.org/abs/2501.16946
- Kulveit, J., et al. (2025). Position: Humanity Faces Existential Risk from Gradual Disempowerment. ICML 2025. https://proceedings.mlr.press/v267/kulveit25a.html