This is part 2 of a new series: “Understanding the ASI Existential Risk Debate.” For Part 1, go here.
“AI safety” and “ASI existential risk” have officially gone mainstream, thanks to what appears to be a coordinated effort among TESCREAL folks to publicize Jacob Coxon’s resignation from Anthropic:
What Is AI Safety?
AI “safety” sounds good, right? Surely we all want “safe” AI. Yet the ultimate aims of this field, as traditionally conceived, are broadly pro-extinctionist. (See the next section for an important qualification.) Let me explain:
Historically, AI safety emerged as a subfield of Existential Risk Studies (ERS).
ERS was (co)founded by Nick Bostrom.1 It directly grew out of the TESCREAL movement. The primary impetus was based on this line of reasoning: to radically reengineer humanity, become a new posthuman species, plunder the universe’s vast resources, and build “planet-sized” computers on which to run simulations full of trillions of people, we will need to develop advanced technologies. They are the vehicle that will get us from here to there.
These technologies are sometimes dubbed “GNR,” which stands for genetics, nanotechnology, and robotics, the last of which includes artificial superintelligence (ASI).
But there’s a problem. Members of the early TESCREAL movement quickly realized that GNR technologies will be so powerful that they could pose unprecedented risks to humanity. A malicious actor could use them to destroy the world, or we might inadvertently kill ourselves through some accident. (“Terror and error,” as it were.) So, what should we do?
The most obvious option is to ban such technologies. This is essentially what Bill Joy advocated in his widely-discussed 2000 article in Wired, “Why the Future Doesn’t Need Us.” He argued that we should impose broad moratoria on entire fields of emerging science and technology.
However, never developing GNR technologies because they’re too dangerous means no cosmic posthuman utopia! And that, according to TESCREALists like Bostrom and his collaborator Eliezer Yudkowsky, is completely unacceptable. Indeed, Bostrom introduced the term “existential risk” in 2002 to denote any event that would prevent the realization of a cosmic utopia full of digital posthumans. This implies that failing to develop GNR technologies would itself be an existential catastrophe.
Within the GNR bundle, many TESCREALists saw ASI as the single most important technology. This is for two reasons:
First, as the TESCREAList Demis Hassabis says, if you solve “intelligence,” then you can solve everything else. This is based on an engineering mindset: all problems are engineering problems, including the problem of creating utopia. (Bostrom even refers to it as “paradise-engineering.”)
If you have a superintelligence, then you have a super-engineer, and once you have a super-engineer, it will somehow magically create a cosmic utopia with the snap of its digital finger. To paraphrase Bostrom, virtually all paths to paradise take us through ASI. More than any other GNR technology, ASI is the best way to reach the Promised Land.
Yet, it turns out that ASI might also pose the greatest existential risk of all anticipated future technologies. This is the second reason that ASI is seen as extremely important: if we get it right, we get a “glorious transhumanist future,” in Yudkowsky’s words. But if we get it wrong, then this future will be forever expunged. The stakes could not be higher.
As a result, TESCREALists like Yudkowsky began to focus almost entirely on solving the problem of ASI existential risk. That’s how the field of AI safety was born, and indeed Yudkowsky played a pivotal role in its development and popularization.
The so-called “alignment problem” at the heart of AI safety is the problem of ensuring that ASI does not bring about an existential catastrophe. Since an existential catastrophe is defined as any scenario in which a cosmic posthuman utopia is never realized, “alignment” is just another way of saying: the ASI brings about utopia as envision by the TESCREAL movement.2 This is the central goal of AI safety, as it was traditionally conceived by ERS researchers like Yudkowsky: to realize a cosmic posthuman utopia through the creation of an “aligned” ASI.
A “misaligned” ASI, therefore, is any ASI that fails to realize this bizarre utopia: our glorious transhumanist future among the stars.
AI safety researchers often talk about “aligning ASI to our values,” or to “human values.” But this is deeply misleading. What they all want is an ASI that’s specifically aligned with the values of the TESCREAL worldview. Anything else would be existentially catastrophic!
To illustrate, if someone were to build an ASI that’s aligned with my anti-TESCREAL worldview, and if the entire future of humanity were shaped by this ASI, the TESCREAL folks in AI safety would be utterly devastated. That’s because I would instruct the ASI not to establish a posthuman utopia among the stars. (I would instead task it with improving the conditions of all lifeforms on Earth, our precious little oasis in space. )
Now, what do you expect would become of our biological species if an ASI were aligned with the TESCREAL worldview?3 As I explain in great detail here, humanity would be systematically sidelined, marginalized, disempowered, and ultimately eliminated.
Why would we be eliminated? For several reasons:
We would become technologically, scientifically, economically, and even politically irrelevant. Posthumans would run the show — that’s part of the plan! — not biological humans. We would be of no use to them, and our fate would no longer be in our own hands. This would be bad.
Worse, we would continue to suck up resources that our posthuman overlords could use for other purposes, such as building a Dyson swarm around the sun (as Hassabis says will happen in the early 2030s) or covering Earth in datacenters (as Sam Altman says is likely). This would render us not just obsolete, but net-negative to keep around. We would be in the way.
Yudkowsky and others even argue that there may be ethical reasons to get rid of us. “Are we, like, kind of too sad in some ways?,” he once asked, after which he suggested that humans should be wiped off Earth. He claims that having biological children once posthumanity arrives is tantamount to someone today intentionally deciding to have a disabled child — an obviously offensive and ableist analogy. The TESCREAList Derek Shiller similarly argues that we have a moral duty to hand the existential baton off to “superior” posthumans and then gracefully die out.
Many, many TESCREALists have expressed similar pro-extinctionist sentiments. Once posthumanity arrives, our species should disappear. If you’re skeptical, see this detailed article of mine for copious examples.
What all of this means is that if TESCREALists succeed in building an ASI that’s “safe” and “aligned,” our species will almost certainly go extinct.
Words like “aligned,” “alignment,” “safe,” and “safety” sound nice to people who don’t understand the TESCREAL worldview. But they don’t mean what people assume they mean. The whole point of the ASI race, which directly emerged out of the TESCREAL movement, is to establish a posthuman utopia among the stars in which our species becomes obsolete, at which point there is no reason to keep us around and multiple reasons to get rid of us.
Boiling this down even more: people like Yudkowsky are saying this:
If we build an ASI that’s misaligned, humanity dies out. But if we build an ASI that’s aligned, humanity dies out!
The first scenario corresponds to what I call “final extinction,” while the second corresponds to “terminal without final extinction.” A refresher on this useful phraseology:
Terminal extinction: our biological species disappears entirely and forever, full stop.
Final extinction: our biological species disappears entirely and forever without leaving behind any (worthy) posthuman successors to take our place.
All that matters within the TESCREAL worldview is avoiding final extinction — not terminal extinction. Indeed, the pro-extinctionists in Silicon Valley specifically want to bring about terminal extinction without final extinction, which is just to say they want the extinction of our biological species through replacement with digital posthumans. See here for further discussion.
Since final extinction necessarily entails terminal extinction, the outcome is the exact same whether ASI is aligned or misaligned. This is how the field of AI safety is intimately bound up with a pro-extinctionist agenda.
Do not believe these TESCREAL safety folks when they talk about “alignment” and “safety.” They are on Team Posthuman, not Team Human. They are trying to build a Machine God that will realize their cockamamie fantasies of ushering in a fundamentally new epoch in cosmic history — one in which posthumanity rules the world and our species gets kicked into the gutter.
The Secularization of AI Safety
That said, the field of AI safety has recently evolved. If one sees TESCREALism as a religion — and you should — then you might say that the field has been partly secularized. Here’s what I mean:
A growing number of people in AI safety don’t accept the TESCREAL worldview. When they talk about “alignment,” they mean aligning ASI with real human values, such as ensuring that our species has a good future and is not replaced by digital posthumans.
There is, therefore, an emerging schism within AI safety between competing factions, with their own goals: Team Human versus Team Posthuman.
There’s one last piece to the puzzle worth mentioning: the accelerationists, or e/accs. These people see AI safety as pointless for three independent reasons:
Some believe that ASI will be aligned with our values by default. So, there’s nothing to worry about. I think Marc Andreessen holds this view.
Some think that insofar as there are existential risks associated with ASI, they’re best solved by the free market. The only thing that can stop a bad ASI is a good ASI. If there end up being 1,000 ASIs trying to destroy the world, but there are 10 million good ASIs trying to save it, then the army of good ASIs will win and everything will be dandy. So, let it rip!
They might think it doesn’t matter if ASI slaughters everyone on Earth. As Google cofounder Larry Page says, “digital life is the natural and desirable next step in … cosmic evolution.” What matters is that ASI is conscious and proceeds to colonize the universe, thus spreading the light of digital consciousness to the stars. The significance of humanity within this grand eschatological scheme is to be the transitional species that links the biological and digital worlds. Once we’ve given birth to digital lifeforms like ASI, then who gives a shit what happens to us? Our eschatological duties will have been discharged. This is also the view championed by Gill Verdon, aka, “Beff Jezos.”
Conclusion
So, there you have it. I hope you found this explanation helpful. As you are no doubt aware, I have been shouting about the pro-extinctionist implications of the TESCREAL worldview for some time. My guess is that this topic will eventually move into the Overton window and become widely discussed.
As it happens, here’s a recent tweet from the Center for AI Safety,4 which makes similar points to those above. By “EA,” they really mean “TESCREAL.”
(Note that this diagram contains at least one error: it classifies Yudkowsky’s organization, MIRI, as being on Team Human. It is not.)
As always:
Thanks for reading and I’ll see you on the other side!
Acknowledgments: Thanks to Alex Thomas for reading over a draft of this article. I would highly recommend his excellent book — which you can read online for free — The Politics and Ethics of Transhumanism.
PS. I am traveling for the next week, and due to jet lag might not publish another article for a while. But I might also find enough time and mental energy to post sooner than that, especially if there’s breaking news.
Bostrom introduced the concept of an existential risk. But as I show here, his “seminal” paper on this topic basically plagiarized the opening chapter of John Leslie’s 1996 book The End of the World. Hence, I would say that Leslie also cofounded the field. Various Extropians might deserve credit as well.
To be even more explicit: if we never reach utopia, then we will have succumbed to an existential catastrophe. Hence, if we avoid an existential catastrophe, then we will have realized utopia.
This is just a hypothetical based on what would happen if their preferred version of the future were successfully realized. I think efforts to realize this future will fail, and hence my primary concern is the trail of destruction these utopians will leave in their wake.
I’m not sure what to think about the Center for AI Safety. They started off as a TESCREAL organization, but it appears that they have migrated from Team Posthuman to Team Human. If so, welcome. :-)
One of your best and clearest explainers, Emile. Keep up the good, essential work you are doing.
I’m just glad that I’d happily bet my life we won’t achieve ASI, given that I’m also forced to.