This is part 1 of a new series: “Understanding the ASI Existential Risk Debate.” For Part 2, go here.
Every claim in this article has been documented and substantiated in other articles of mine, complete with citations. If you find yourself skeptical of some particular claim about these ideologies, or about specific people, either read my articles or ask me about it in the comments section and I’ll share the relevant citation.
I’d like to answer, in writing, two questions that journalists have been asking me the past few days:
Do people like Jacob Coxon, Dario Amodei, and Sam Altman sincerely believe that ASI (artificial superintelligence) could bring about human extinction?
Why won’t they just slow down? As David Sacks said on X, they don’t need the government to voluntarily slow down to, you know, not kill everyone on Earth.
The Whole Discourse Is Fucked
The first point I’d make is that the whole public conversation about “human extinction” is fundamentally misleading. When people who espouse the Silicon Valley worldview — which is to say, the TESCREAL worldview — talk about “human extinction,” they don’t mean what the overwhelming majority of people think they mean.
Most people interpret this as saying: We must save the human species — biological humans. That’s not what they mean. If our species were to die out, this would constitute terminal extinction. But there’s a different kind of extinction that the TESCREALists actually care about: final extinction, whereby we die out and leave behind no “worthy” posthuman successors to take our place. (When I spent 10 years in the TESCREAL movement becoming the 6th most-published contributor to the academic literature on existential risks, every time I wrote “human extinction” I meant “final human extinction.”)
Terminal extinction: our species disappears entirely and forever, full stop. (This is what I care about avoiding today, after leaving the TESCREAL movement.)
Final Extinction: our species disappears entirely and forever without leaving behind any “worthy” posthuman successors. (This is what they care about avoiding.)
This distinction really, really matters. There’s a massive difference, they believe, between us dying out and leaving behind successors, and us disappearing without doing so.
The whole point of the Silicon Valley worldview is to build an ASI that will bring about a cosmic posthuman utopia. They want it to transform us into digital posthumans who can then colonize the universe. (Some just want ASI to simply replace us like an invasive species let loose on an island. See this article for details.)
The only type of extinction that would permanently prevent the creation of a cosmic posthuman utopia is final extinction. The only reason we should avoid terminal extinction — the disappearance of our species — is that if terminal extinction were to happen tomorrow, it would entail final extinction.
But if terminal extinction were to happen after “worthy” digital posthumans arrive, then it wouldn’t entail final extinction. Since final extinction is the only way a cosmic utopia would be permanently expunged, TESCREALists care about terminal extinction only insofar as it would entail final extinction.
Once posthumanity is here, who gives a shit about what happens to our species? As I write in great detail here, many of these people hold the “replacement view” (also called “pro-extinctionism”), according to which our species should die out once posthumanity arrives.
To emphasize, many, many hold this view. Yudkowsky holds this view. So does Sam Altman, Larry Page, and Gil Verdon — and so on, and so forth. Using my terminology from above, what this amounts to is that these people want terminal extinction without final extinction — which is just to say, they want our species to stick around long enough to create or become our posthuman successors, and then to kick the bucket.
There are two reasons for this: (1) once posthumans rule and run the world, Homo sapiens will become obsolete and irrelevant in every respect: technologically, scientifically, economically, and even politically. We will serve literally no purpose. (2) Even worse, we will continue to suck up resources that our “superior” posthuman successors could use “more efficiently” — for things like covering Earth in data centers (which Altman says is likely), building a Dyson swarm around the sun (as Demis Hassabis says will happen starting in the early 2030s), and spreading the light of digital consciousness to the stars (which all of them endorse).
There may even be “ethical” reasons for getting rid of us. As Yudkowsky declares, “Are we, like, kind of too sad in some ways” to keep around? He adds that once posthumanity is here, continuing to create biological humans would be like someone right now intentionally deciding to have a child with disabilities. (Yes, this is outrageously ableist. But then again, he is a eugenicist of the most radical kind.1)
***
So, this is why the conversation about “human extinction” is completely fucked. Making matters worse, the TESCREAL academic literature explicitly defines “human” as our species plus whatever posthuman successors we might have. This immediately implies that our species could completely die out next year without “human extinction” having happened. As long as there are posthumans to take our place, “humanity” will persist and “human extinction” will have been avoided. When they talk about “preserving humanity,” they really mean “securing posthumanity.”
What does this all mean? That if the TESCREAL extremists with their bizarre Silicon Valley fantasies about a cosmic posthuman utopia were to succeed in building an ASI that’s “aligned” with the values of TESCREALism — and that’s their goal — then human extinction would almost certainly result, because we’d be pushed out of existence by digital posthumans and ASI, exactly as Altman predicts.2
Amodei presents this more gently: he talks about ASI radically augmenting our cognition, extending lifespans, and enabling us to “transcend our biology” through mind-uploading. But that’s essentially no different from Altman’s vision. Here’s what Holly Elmore, who has personally known people at Anthropic, says about this:
With EAs, they were so interested in superintelligence and a lot of the people who cared about AI safety were there because initially they wanted AI to do something like usher in heaven through the Singularity, and they were trying to make sure that that would happen instead of a bad outcome that could happen with superintelligence. …
Many people at Anthropic believe that they might be making the next species to succeed us. That maybe humans don’t live after that, and so it’s really important to give Claude good values, because of that, because we need to make our own values persist.
Despite their screams about “avoiding human extinction” if ASI is “misaligned,” most of them actually advocate for human extinction, where “human” here specifically means “our species.” Put another way: if what one cares about is avoiding terminal extinction — and I do! — then these people are pro-extinctionists. (My view, to be clear, is that our biological species is what matters, for reasons I’ve outlined elsewhere.)
The media needs to understand this, fast. What the TESCREAL doomers are worried about is an ASI that wipes out our species, thus foreclosing the possibility of us becoming or creating “worthy” successors: posthumans. What they want is an ASI that enables us to become or create “worthy” successors that usurp humanity.
In the first case, the outcome would be final extinction (and hence terminal extinction, since final extinction entails terminal extinction). In the second case, the outcome would be terminal extinction but not final extinction. Once again, the goal for many if not most of them is terminal without final extinction.
Do They Sincerely Believe?
Now we can be more precise with the first question: do Coxon, Amodei, Altman, and the others sincerely believe that “misaligned” ASI could bring about final human extinction?3 Or are they just using extinction worries to further concentrate power: “We’re building this thing that could kill everyone (and erase utopia), and we hold the only solution, so give us more money and power!”
These are not mutually exclusive! I spent years in the TESCREAL movement, and can confirm that everyone is genuinely scared shitless about the possibility of a misaligned ASI murdering everyone. They aren’t lying when they say that an existential catastrophe caused by ASI is a real possibility.
Note that, because these people hold a “further-loss view” in the ethics of human extinction (see here for explanation), they believe the worst part of our extinction by far wouldn’t be the death of 8.2 billion people. Rather, the worst part of our collective demise is that it would foreclose the realization of literally “astronomical” amounts of future value if we were to create an “aligned” ASI, replace humanity with “worthy” posthumans, colonize the universe, and build “planet-sized” computers full of trillions of digital posthumans. As Peter Singer, Matthew Wage, and the cofounder of longtermism write:
One very bad thing about [final] human extinction would be that billions of people would likely die painful deaths. But in our view, this is, by far, not the worst thing about human extinction. The worst thing about human extinction is that there would be no future generations.
That said, I have no doubt that many AI CEOs realize the “extinction risk” narrative also gives them more power. And that’s what they want. Longtermists have literally talked about trying to acquire the greatest possible fraction of the world’s wealth and power.
These people are, essentially, religious extremists: they believe fervently in their vision of the future; they believe that literally “astronomical” amounts of value hang in the balance re: getting ASI right; they believe they have special access to fundamental “truths” about what the future ought to look like; they think the masses are too stupid to understand the stakes, and hence that democratic input from the rest of humanity is unnecessary and irrelevant.
This is one of the dangers of worshiping “rationality.” Those who see themselves as having superior rationality assume they have a fully objective “view from nowhere.” It’s not a point of view, it’s the point of view of objective truth, so to speak. Why do you need to ask people in India, Nigeria, China, Bolivia, etc. what the future should be like when you can clearly see what it should look like from your special, objective view from nowhere?
Because they believe that realizing a cosmic posthuman utopia is so important, they will stop at nothing to acquire as much wealth and power as possible to ensure this comes to pass. And warning about “ASI extinction risk” helps them acquire the wealth and power they so desperately want, for what they see as fundamentally ethical (utilitarian) reasons (“maximize value in the cosmos!”).
Do not underestimate how power-hungry these people are. As someone who spent a decade in the TESCREAL movement, I can personally attest to many conversations behind closed doors about how to do this!
Why Not Stop?
If the AI CEOs think that misaligned ASI might kill everyone and, in the process, permanently erase our cosmic posthuman future, why don’t they just stop? Why continue building ASI?
There are several reasons. First, many of these people suffer from extreme death anxiety. They think that if we build an ASI that is “aligned,” it will grant them eternal life, enabling them to upload their minds to computers, etc. This is one reason.
Second, they all love expected value (EV) theory. Given the supposed “astronomical” value of cosmic utopia, even a relatively high probability of total annihilation in the future could still justify moving forward. Imagine there’s a 20% chance of your plane going down and you dying. (This is Amodei’s estimate of final extinction in the near future from misaligned ASI.) Now imagine that if it doesn’t go down, you get to become an immortal super-being who literally owns a galaxy forever in the literal heavens with all your best friends. The immense “value” of this positive outcome makes an 80% chance of surviving worth taking.
Nick Bostrom himself — a TESCREAList — recently argued that we should proceed with ASI even if there’s up to a 97% chance of universal death, given the (totally unfounded) assumption that in the 3% case, the aligned ASI would grant everyone more than a millennium of extra life.
Third, you have to understand how these people define an “existential risk” or “existential catastrophe.” Introduced specifically in the context of libertarian transhumanism, it refers to any event that would permanently prevent us from creating a cosmic posthuman utopia. Final human extinction is the most obvious type of existential risk/catastrophe. But there are an infinite number of existential catastrophes in which our species, Homo sapiens, survives.
To illustrate: imagine that we somehow establish world peace. Nuclear nations abolish their arsenals. There are no more wars, climate change is reversed, and the ecological situation is radically improved. All diseases are cured, and people live wonderful lives into their late 90s. Loneliness disappears, everyone is happy, and we all live in sustainable eco-technological villages amid meadows and gardens watching the sunset every night. After 1 billion years of this, we collectively decide to “die with dignity” by allowing our species to fade away as the sun becomes a Red Dwarf that renders Earth uninhabitable. Despite a rocky start to civilizational history, everyone around the world is at peace with how well our story turned out.
This would be an existential catastrophe whose badness is on par with final human extinction next year in some kind of devastating bloodbath. Why?
Because in both cases we will have failed to realize more than a teeny-tiny fraction of all the cosmic “value” that could have otherwise existed within our future light cone. And it’s the realization of this future “value” that ultimately matters — value for whom? That’s the wrong question, they’d say. This “value” is just intrinsically valuable, from “the point of view of the universe.” The more “value” the universe contains across space and time, the better it becomes, full stop. (If you find this unsatisfying, then join the club!)
This is the real reason the AI companies won’t stop: ASI is almost certainly necessary to create a cosmic posthuman utopia. As Bostrom puts it, “all the plausible paths to a really great future involve the development of machine superintelligence at some point.”
You’ll hear the AI companies give different reasons like:
We can’t stop because we’re in a race with other US companies. These other companies are less responsible than us. If they reach the ASI finish line first, we might all die (final extinction). If we reach it first, then at least we have a chance.
We can’t stop because we’re in a race with Chinese companies. If China wins, then it will establish a global authoritarian communist state.
But think about what underlies these excuses. Both are reducible to the TESCREAL goal of creating utopia.
With respect to (1), the companies are saying that if another, less responsible company creates ASI first, it will probably be misaligned, and misaligned ASI would be very, very bad because it would destroy utopia.
With respect to (2), the companies are saying that if China were to build an ASI aligned with its political ideology, this would also prevent the realization of the utopian TESCREAL worldview. Would China proceed to reengineer humans to create a new species of digital posthumans, who then go out and plunder the universe’s vast resources like a good utilitarian and capitalist? Maybe not — probably not. No one in China, so far as I know, espouses the TESCREAL worldview. Their goals are different, and that poses a threat to the TESCREAL movement.
The ultimate reason these companies won’t stop is because ASI is believed to be necessary to create utopia, and these people desperately want utopia as soon as possible. Even the doomiest doomers want ASI — specifically, “aligned” ASI — as soon as possible. As Yudkowsky’s Machine Intelligence Research Institute puts it, borrowing the language of longtermism,
we remain committed to the idea that failing to build smarter-than-human systems someday would be tragic and would squander a great deal of potential [i.e., posthumans living in computer simulations4]. We want humanity to build those systems, but only once we know how to do so safely.
***
This is why I have been shouting for years that, to successfully stop the ASI race, we need to address the underlying ideological drivers that are pushing it forward. So long as those drivers remain in place, influential and powerful, championed by power-hungry billionaire oligarchs, the ASI race will move forward.
The question of “pausing” or “slowing down” the race specifically concerns the race between AI capabilities research and AI safety research. The former aims to build an actual ASI, while the latter aims to ensure that it’s “aligned” with the TESCREAL worldview. The point of a slow-down would be to hit the breaks on capabilities research so that safety research can catch up, because this is how they maximize the probability that ASI is going to help us realize a cosmic posthuman utopia.
This isn’t about not building ASI. That would be existentially catastrophic. And as I argued above — and explain in immense detail here — if these TESCREAL extremists succeed in building an “aligned-with-TESCREALism” ASI, the likely outcome would be terminal extinction: the expungement of our species.
If there is an extinction risk here, it arises from the successful realization of the TESCREAL vision. This is what everyone is missing.
When these people warn about “human extinction,” they’re talking about final human extinction, whereby we don’t get to create or become “worthy” posthumans who will then replace our species. The probable outcome of ASI creating a new species of posthumans is that our species will be sidelined, marginalized, disempowered, and ultimately eliminated. As Ray Kurzweil once put it, if you choose not to become posthuman “you won’t be around for very long to influence the debate.”
Conclusion
So, TESCREAL folks really are worried about ASI causing final extinction. But they also no doubt understand that shouting about “extinction risk” will increase their power to control the situation — to control, ultimately, how the entire future of (post)humanity unfolds in the cosmos.
And they won’t stop trying to build ASI because they think it’s necessary to realize their techno-utopian fantasies. At most, they’ll slow down a bit to let safety research catch up, where the central focus of “AI safety” is to guarantee a cosmic utopia rather than total annihilation (where a cosmic utopia would probably mean the disempowerment and elimination of our species).
Finally, they exploit the ambiguity of the term “human extinction” to make their views sound infinitely more reasonable than they actually are. Members of the public who know nothing about the TESCREAL movement hear Altman and the others warning about “human extinction” and think: “I also don’t want our species to die out. I guess we’re on the same side.”
No! False! That is not what they mean. Again, Altman is explicit that the only scenario in which “we survive” is by becoming digital posthumans — something Altman himself has already taken steps to become by signing up with Nectome to have his brain digitized.
I am begging the media to investigate these issues more; I am pleading with them to make the effort to understand the Silicon Valley worldview that nearly every major figure in the ASI race embraces. This will not end well for humanity even if the ASI race is slowed down and an “aligned” ASI is successfully built.
Thanks for reading and I’ll see you on the other side!
Immense thanks to Alexander Thomas, author of the brilliant book The Politics and Ethics of Transhumanism (which you can read for free here), for thoughtful feedback on this post. Any errors are my own.
I.e., he’s a transhumanist. Whereas the eugenicists of the 20th century wanted to “perfect” humanity, transhumanists want to completely “transcend” it. That is eugenics on steroids.
Hence, “alignment” — as pursued by TESCREAL folks in the AI safety field, which originated as a branch of the TESCREAL movement — doesn’t merely mean “avoiding human extinction.” It’s fundamentally about a positive rather than a negative: realizing a posthuman utopia among the stars.
Two things I wanted to say:
1. This isn't me disagreeing with you, Émile, but while you are correct that many, many people who subscribe to the TESCREAL worldview do genuinely believe that a "Misaligned AI" could/will be coming soon and WILL kill all of Humanity...I think it's still important to remember that these people, from Yud to Altman to Sutskever and more, are most likely suffering from some kind of AI Psychosis. You say it yourself; they're "Religious Extremists". There's a LOT of people who also believe in the Rapture. And yet the Rapture has never happened despite the crying, preaching, and constant media attention from these Religious Extremists. I just think that while this talk of Misaligned AI is scary...we have to remember (Which you already do :) that we are hearing this from people who are Techno-Religious Extremists...Who also think Humanity should die. That's bad, but they don't need an AI God to do that; they can just kill us with pollution from Data Centers.
2. All this talk with Jacob Coxon has got me thinking, as I usually do, about something in particular: Does anyone remember Daniel Kokotaijlo, the guy who made AI2027? He and Jacob Coxon are...basically the same thing. Sure, Kokotaijlo didn't make a Tweet saying AI could kill us by the end of the decade and that AI Companies need to "Lock in and make a Kill Switch", but he did make AI2027, which, while terrible and mostly a hypothetical Sci-Fi, anti-China PowerPoint, was talked about for a while. Even JD Vance "claimed" to read it. Kokotaijlo also got big because he claimed not to have signed an NDA with OpenAI so as to bash them (Still retained equity, though).
My point is, why has the news reacted this way with Coxon, when Kokotaijlo literally did the same thing LAST YEAR!?
Great post as always, Émile.
"They think that if we build an ASI that is “aligned,” it will grant them eternal life, enabling them to upload their minds to computers, etc."
This bit confuses me. Why on earth would you assume that uploading ones mind onto a computer would grant you eternal life?
It seems based on some notion that conciousness is a form of software. But doesn't seem an awful lot of evidence for that. Speculative at best.
Obviously, this is assuming that such uploading is even possible.