Nick Bostrom Took Many of His "Best" Ideas from Other People
An investigation with examples so you can decide for yourself. (7,300 words)
Nick Bostrom is the former director of the Future of Humanity Institute (FHI) at Oxford. His publications on transhumanism, longtermism, and artificial superintelligence (ASI) have had a major impact on the TESCREAL movement, and directly inspired the current race to build ASI.
I’ve followed his work since about 2005, and it’s been clear to me for some time that Bostrom considers himself to be a towering intellect of cosmic-historical importance. He is not shy when boasting about his supposed accomplishments, even quoting on his website an article that describes him as the “Swedish superbrain.” For many years, he bragged that — to quote his profile on Edge.org — “his performance as an undergraduate set a national record in Sweden,” though he backed away from this after Simon Knutsson pointed out that there is no evidence to support such a claim. (See the “Potentially Dishonest Self-Promotion” section of this article.1)
According to a TESCREAList named Diego Caleiro, as Caleiro departed FHI many years ago, Bostrom turned to him and “made a gesture like Atlas, holding the world above his shoulders, and quasi letting the world fall, then readjusting.” Caleiro writes that “while funny, I also understood the obvious connotation of how tough it must be to carry the weight of the world like that.”
Yet, because I’m so conversant with the foundational literature of the TESCREAL movement, I’ve noticed that a large number of the “original,” “novel,” even (supposedly) “groundbreaking” ideas attributed to Bostrom actually originated with other thinkers. As early as 2018, I was posting about this on social media and privately sharing examples with colleagues — some of whom were rather shocked to see Bostrom’s work juxtaposed with that of others (from whom he borrowed without sufficient attribution). I wrote to one colleague that Bostrom’s “genius” seems to be taking ideas from other theorists and repackaging them in his own language such that they’re just different enough for him to avoid being accused of intellectual plagiarism. His oeuvre reminds me of the line, often misattributed to Picasso: “Good artists copy; great artists steal.”
Here are just a few examples. Don’t take my word for it — decide for yourself:
1. Existential Risk Scenarios
1.1 Bostrom’s 2002 Paper
Perhaps the most “famous” paper written by Bostrom is his 2002 “Existential Risks: Analyzing Human Extinction Scenarios and Related Hazards.” I first read this circa 2005 and was struck by its breadth and originality. Bostrom offers a technical definition of “existential risk,” which denotes any event that would prevent us from creating a (utopian) posthuman civilization. (Hence, the term is intimately bound up with his version of transhumanism.)
About half of the paper is a detailed classification of different types of existential risk scenarios, such as2:
Deliberate misuse of nanotechnology and a nanotechnology accident (such as the gray goo scenario).
Nuclear holocaust.
Misaligned superintelligence.
Engineered pandemics.
Natural pandemics.
Physics disasters, e.g., vacuum decay.
Asteroid or comet collisions.
Runaway climate change.
Fundamentalist religious or ecological movement that halts technological development.
An extraterrestrial civilization kills us.
“Something unforeseen,” because “it would be foolish to be confident that we have already imagined and anticipated all significant risks.” (Take note of the wording.)
Examples of the above include “solar flares, supernovae, black hole explosions or mergers, gamma-ray bursts, galactic center outbursts, supervolcanoes,3 loss of biodiversity, buildup of air pollution, gradual loss of human fertility, and various religious doomsday scenarios.” (Take note of each of these threats.)
Resource depletion or ecological destruction.
Totalitarianism.
Insurmountable technological hurdles (that prevent us from developing advanced technologies).
Our simulation gets shut down. (Lolz)
Dysgenic pressures, whereby less “intelligent” people outbreed their more “intelligent” peers. (Bostrom is a eugenicist.)
He then discusses the Fermi paradox, anthropics, observation selection effects, and the Doomsday Argument. Taking these in order:
The Fermi paradox (which is neither a paradox nor Fermi’s!) points out that it’s surprising we don’t see evidence of technologically advanced civilizations given the age and size of the universe. Hence, Bostrom writes that
there must be (at least) one Great Filter — an evolutionary step that is extremely improbable — somewhere on the line between Earth-like planet and colonizing-in-detectable-ways civilization. If the Great Filter isn’t in our past, we must fear it in our (near) future. Maybe almost every civilization that develops a certain level of technology causes its own extinction.
Observation selection effects concern the question of “what we should expect to observe given some hypothesis about the distribution of observers in the world.” For example, you might think that a large asteroid striking Earth is improbable because we haven’t witnessed one strike Earth in the past 300,000 years (the period during which our species has existed). But if an asteroid had struck Earth during this time, we wouldn’t be here to talk about it having happened. Hence, we will only ever find ourselves in worlds where asteroids haven’t struck in the recent past. This means that the evidence available to us will be biased in certain ways.
The Doomsday Argument is an a priori argument for why human extinction is more probable than we’d otherwise expect — that is, based on empirical observations. The reasoning is interesting: basically, it’s more likely that we’d find ourselves in a large population at the end of history than in the much smaller population of people long ago, when human history was just beginning. We should thus increase our probability estimates of a secular apocalypse happening soon.
1.2 John Leslie’s 1996 Book
Again, I was extremely impressed with Bostrom’s paper when I first read it. It was original and erudite — even polymathic.
But then I read the introductory chapter of John Leslie’s 1996 book The End of the World and was utterly shocked by how similar Bostrom’s paper is to it. (And, as I’ll explain in a moment, we know that Bostrom was very familiar with Leslie’s book.) For example, although Leslie doesn’t use the term “existential risk,” the central topic of his tome is human extinction — the most obvious type of existential risk. In the introduction, he lists scenarios like the following that could potentially have annihilatory consequences4:
A disaster from nanotechnology (including the “gray goo” scenario).
Nuclear war.
Computer-related disaster (e.g., computers replacing us).
Engineered pandemics (i.e., “a disaster from genetic engineering,” including “green scum” and altered microorganisms that could render women infertile).
Natural pandemics.
Physics disasters, e.g., vacuum decay.5
Asteroid or comet collision.
Runaway climate change.
Religious or philosophical ideologies that prevent humanity from surviving, including Schopenhauerian pessimism, ethical relativism, negative utilitarianism, and “person-affecting” ethical theories.
Extraterrestrials kill us.
“Something-we-know-not-what,” because “it would be foolish to think we had foreseen all possible natural disasters” and “we cannot possibly imagine every single danger which technological advances might bring with them.” (Sound familiar?)
Nearby supernova or black hole merger, volcanic supereruptions, pollution, ozone depletion, etc.
Extreme ice age.
Breakdown of complex systems.
Unwillingness to have children.
Biological or chemical warfare.
Some other kind of technological disaster.6
These lists are clearly very similar. My impression at the time of first reading Leslie was that the bulk of Bostrom’s paper is little more than a summary of Leslie’s opening chapter. Bostrom even uses similar language. For example, in adding “unknowns” sections at or toward the end of each list (exactly as Leslie does), he writes: “It would be foolish to be confident that we have already imagined and anticipated all significant risks.” Compare that with what Leslie says: “It would be foolish to think we had foreseen all possible natural disasters … [and] every single danger which technological advances might bring with them” (italics added to both quotes).
Even more, Leslie then proceeds to discuss the Fermi paradox, anthropics, observation selection effects, and the Doomsday Argument (which he partly developed) — exactly as Bostrom does, except years before Bostrom wrote his paper.
Some humor: the Doomsday Argument should not be confused with the Doomsday List, which is — I kid you not — a list of endangered lighthouses. There’s also the “Doomsday rule,” a mnemonic device “for working out the day of the week corresponding to any given date.” So much “doom,” so little doom.
These similarities aren’t a coincidence: Bostrom wrote his PhD dissertation on the topic of anthropics, and relied on Leslie’s 1996 book — mentioning Leslie’s name 135 times and citing the book throughout. Bostrom even thanks Leslie in the Acknowledgments section.
So far as I can tell, the only thing that’s nontrivially original about Bostrom’s 2002 paper is the existential risk concept, which he introduced to capture an idea that was, as it happens, already being discussed by his online Extropian buddies. Whereas Leslie focuses specifically on human extinction, Bostrom recognized that there are a range of survivable scenarios that would also prevent us from creating a posthuman utopia full of “astronomical” amounts of value. That’s not nothing, but it’s not a lot, either.7
2. Where Are They?
In 2008, Bostrom published an article in the MIT Technology review titled “Where Are They? Why I Hope the Search for Extraterrestrial Life Finds Nothing” — which he presumably got paid $1 to $2 per word for.8 His central thesis is that “I hope that our Mars probes will discover nothing. It would be good news if we find Mars to be completely sterile. Dead rocks and lifeless sands would lift my spirit.” He continues:
Conversely, if we discovered traces of some simple extinct life form – some bacteria, some algae – it would be bad news. If we found fossils of something more advanced, perhaps something looking like the remnants of a trilobite or even the skeleton of a small mammal, it would be very bad news. The more complex the life we found, the more depressing the news of its existence would be. Scientifically interesting, certainly, but a bad omen for the future of the human race.
Why does he say this? Given the Fermi paradox, there must be at least one Great Filter that prevents life from developing far enough to colonize the universe and make its presence known. This Great Filter could be behind us or ahead of us.
For example, it could be that the step from non-life to single-celled living organisms (a process called abiogenesis) is extremely unlikely. If so, then we have gotten very lucky — and the mystery of why we don’t see technologically advanced civilizations flitting about the galaxy, given the age and size of the universe, makes perfectly good sense. Or it could be that the step from single-celled organisms to multicellular lifeforms is extremely improbable, or that the subsequent emergence of big-brained creatures like us is highly unlikely.
Alternatively, it could be that all of these steps are fairly probable — i.e., none constitute a Great Filter. Instead, the universe is teaming with big-brained creatures that, just like us, build industrial civilizations, destroy their environments, invent nuclear weapons, run dangerous physics experiments, etc. and then promptly destroy themselves before having initiated a colonization explosion. If that’s the case, then the Great Filter lies ahead of us, meaning that we’ll soon go extinct due to rogue superintelligence, climate breakdown, nuclear war, etc.
What does this imply? It implies that finding evidence of simple lifeforms in the universe — including on our planetary neighbor, Mars — will shift the probability of the Great Filter being behind us to it being ahead of us. If it turns out that simple lifeforms are common, then the Great Filter probably isn’t the step of abiogenesis. That means that the probability of the Great Filter being in our (near) future rises — which is, of course, very bad news for us! It means we might all die in some kind of extinction event in the coming years or decades.
That’s Bostrom’s ostensibly original point. Throughout the article, on my reading, he repeatedly presents this idea as his own. (Read it for yourself and let me know what you think.) “So this is why I’m hoping,” he concludes, “that our space probes will discover dead rocks and lifeless sands on Mars, on Jupiter’s moon Europa, and everywhere else our astronomers look” (italics added). “It would keep alive the hope for a great future for humanity.”
Yet this isn’t his idea at all. It’s clearly included in the 1998 article titled “The Great Filter — Are We Almost Past It?” by Robin Hanson, which introduced the idea of the Great Filter. Hanson writes:
While all of these stories [about how each evolutionary step leading from non-living matter to space colonization could happen] are at least minimally plausible, our main data point implies that at least one of these plausible stories is wrong — one or more of these steps is much more improbable than it otherwise looks. If it is one of our past steps, such as the development of single-cell life, then we shouldn’t expect to see such independently evolved life anywhere within billions of light years from us. But if it is a step between here and a choice to explode that is very improbable, we should fear for our future. At the very least, our potential would have to be much less than it seems. Optimism (as defined here) regarding our future is directly pitted against optimism regarding the ease of previous evolutionary steps. To the extent those successes were easy, our future failure to explode is almost certain.
In other words, if the steps from non-living matter to single-cells, from single-cells to multicellular beings, and from multicellular beings to big-brained creatures are “easy” — if there is no Great Filter making them highly improbable — then we should fear for our future, because it implies that the Great Filter must be ahead of us. Since finding life on Mars would suggest that there isn’t a Great Filter in our evolutionary past, then suddenly the probability that it lies in our future rises.
I read Bostrom’s article before Hanson’s, and once again I believed that Bostrom was articulating a novel insight. But he wasn’t. Yet Bostrom only once mentions Hanson in the MIT Technology Review paper — not to acknowledge that Hanson made the same point, but to parenthetically attribute the term “Great Filter” to him. Bostrom writes: “I borrow this term from Robin Hanson, an economist at George Mason University.”
Meanwhile, over on Bostrom’s personal website, he posted a slightly different version of the article that includes a footnote that says: “I borrow this term from Robin Hanson’s ‘The Great Filter — Are We Almost Past It?’ … a paper which presents an argument similar to the one expounded here.”
If you were to miss that brief footnote in the version posted on his personal website — the only place where he acknowledges the unoriginality of his thesis — then you’d naturally think Bostrom’s article is making a novel point. Readers of the MIT Technology Review most likely did, as did I when I first read it. Turns out Bostrom was just echoing Hanson’s idea.
And for this, he received somewhere between $4,692 and $9,384 (again, $1 or $2 per word), assuming that MIT Technology Review paid writers the same back in 2008 as it does now. That’s a lot of money for rehashing someone else’s point without even noting, as he does in the personal-website (but not officially published) version, that Hanson “presents an argument similar [nay, identical in essence] to the one expounded here.”
3. Superintelligence
In 2014, Bostrom published a surprise bestseller titled Superintelligence: Paths, Dangers, Strategies. It offered a comprehensive overview of the argument for why ASI could pose a dire existential risk. Indeed, Bostrom contends that the “default outcome” of ASI is “doom,” because figuring out how to “align” ASI with the “values” of humanity (by which he really means the values of the TESCREAL worldview) poses enormous technical and philosophical challenges.
This book was hailed as an important contribution to the study of ASI existential risk. Elon Musk endorsed it, as did Bill Gates. Sam Altman called it “the best thing I’ve seen on this topic. It is well worth a read.”
Yet it turns out that much of the book is little more than a summary of ideas outlined by Bostrom’s TESCREAL colleague Eliezer Yudkowsky (who was himself building upon earlier writers like IJ Good, Verner Vinge, etc.). As Ben Goertzel writes in a review of the book, “nearly all the core ideas of Bostrom’s work appeared previously or concurrently in Yudkowsky’s thinking.” In a section titled “Scratch the Surface of Bostrom, Find Yudkowsky,” he reports that
my first reaction, as I paged with curiosity through Bostrom’s gloomy, erudite tome, was to be struck by the remarkable similarity between the arguments, and the general attitude presented, and the perspective on the future of AI put forth by Eliezer Yudkowsky over the last 15 years or so. Yudkowsky has not (yet) written any single document as thorough and carefully-wrought as Superintelligence. But in mailing list and blog posts and online essays over the years, he has articulated a perspective very close to Bostrom’s, and in many ways possessing more depth and nuance.
Goertzel adds that
with only mild oversimplification, one might say that Bostrom’s Superintelligence is a triumphal work synergizing Bostrom’s communication and analytical ability and Yudkowsky’s creative ideas. Of course, this oversimplification isn’t quite fair to Bostrom, who contributed a lot of detailed ideas to his book as well. But as for the Big Ideas in Superintelligence, it really seems that every one was articulated at some point previously by Eliezer Yudkowsky.
So, Bostrom gets some credit for “a lot of detailed ideas,” but the most significant ideas in the book came from Yudkowsky — pictured below.
4. Instrumental Convergence vs. AI Drives
It’s not just Yudkowsky that Bostrom drew from when writing about ASI existential risk. A central premise of the argument for ASI x-risk is what Bostrom calls the “instrumental convergence thesis.”
Bostrom first delineated this thesis (so far as I know) in his 2012 article, “The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents.” It states that agents with a sufficiently high level of “intelligence” will predictably pursue a finite set of intermediary or “instrumental” goals, given a wide range of final goals (i.e., the ultimate goals the ASI wants to achieve). Such instrumental goals include:
Self-preservation: ensuring that it stays alive, because an agent can’t reach its final goals if it perishes.
Goal-content integrity: ensuring that it’s goals stay the same over time, because an agent at T1 can’t reach its final goals if it allows those final goals to change at some later time T2.
Cognitive enhancement: improving its cognitive abilities, including its rationality, as this will help it find new and better ways to attain its final goals.
Technological perfection: create ever-more powerful technologies, as this will help it bend the world toward realizing its final goals.
Resource acquisition: acquire as many resources as possible, as resources are necessary for achieving most final goals that it might have.
Whether the final goal of an ASI is curing cancer, calculating the 100-millionth digit of pi, or converting the entire universe into data centers, we can predict — the claim goes — that in trying to achieve such goals, the ASI will attempt to preserve itself, prevent its goals from being altered, improve its cognition, perfect technology, and acquire maximal resources. This is the instrumental convergence thesis.
Once more, I came away from this paper after reading it 10+ years ago thinking that it was yet another publication of notable originality from Bostrom. Only later did I notice a footnote hidden at the bottom of one of the pages that says a guy named “Stephen Omohundro has written two pioneering papers on this topic,” one being “The Basic AI Drives,” published in 2008 — four years before Bostrom’s paper and six years before Superintelligence. Bostrom goes on to say that he doesn’t like Omohundro’s “AI drive” terminology. He thinks “instrumental goals” is better. He then adds at the very end of this footnote that his “treatment here also differs from that of Omohundro in some other more substantial ways, although the underlying idea is the same.”
I would agree that “the underlying idea is the same,” because it obviously is. For example, in Omohundro’s paper, he identifies the following as AI drives (i.e., instrumental goals) that any sufficiently intelligent system will almost certainly pursue:
“A strong drive toward self-preservation.” Again, if the AI ceases to exist, it won’t be able to achieve its final goals, which means it will take steps to ensure its survival.
“AIs will try to preserve their utility functions,” which Bostrom calls goal-content integrity.9
Self-improvement, such as developing “more efficient algorithms, more compressed representations, and better learning techniques” to augment its cognitive abilities. This also means the AI systems will try to make themselves more rational. In other words, cognitive enhancement.10
“AIs will want to acquire resources and use them efficiently,” given that “all computation and physical action requires the physical resources of space, time, matter, and free energy.”
Bostrom’s list isn’t entirely identical with Omohundro’s. But it’s very close. Yet, when TESCREAL folks talk about the topic, almost no one cites Omohundro. Some aren’t even aware that he published an article strikingly similar to Bostrom’s. Rather — as with all the examples above — Bostrom gets most of the credit, as if he was the first to hit upon these ideas.
5. Letter From Utopia Vs. Letter to Mother Nature
Back in 2008, I emailed one of the founding members of the Extropian movement. (Bostrom was once an Extropian.) In the email, I mentioned Bostrom’s essay “Letter from Utopia,” as I was using it for an art project. The Extropian liked my art project but said:
A suggestion is to read Max More’s “Letter to Mother Nature” [1999] written years before Nick grabbed onto the idea and then did his own version with no credit or mention of Max. … I just wish that you were using the original idea and the original letter because that would give your [work] integrity and honesty.
Bostrom published “Letter from Utopia” in 2008 (though he claims to have first circulated a draft in 2006), while More read his “Letter to Mother Nature” at an Extropianism conference in 1999. Not only are the titles similar, but the content overlaps considerably. For example, More lists seven “amendments” to the “human constitution,” namely:
Overcome aging and death.
Expand perceptual range (new senses).
Enhance cognition.
Supplement the neocortex with a “metabrain,” i.e., a “distributed network of sensors, information processors, and intelligence.”
Use genetic engineering to control biological and neurological processes.
Refine emotions and improve rationality.
Control our biochemistry and integrate technology into our bodies.
The goal is for “these amendments to our constitution [to] move us from a human to a posthuman condition as individuals.” Along exactly similar lines, Bostrom’s “letter” is written by a fictional posthuman who declares that we could become posthuman if only we pursue “three fundamental transformations,” namely:
Overcome aging and death.
Enhance intelligence and perceptual range. (“And then ask yourself, what other harmonies are there in the air, that you lack the ears to hear? What vaults of value are you witlessly debarred from, because you lack the key sensibility? … Your brain must grow beyond the bounds of any genius of humankind, in its special faculties as well as its general intelligence …”)
Elevate well-being. (Maximize our ability to experience pleasure. Increase our rationality: “All emotions have a function. Prune and weed carefully lest you accidentally reduce the fertility of your plot.”)
Not once does Bostrom cite More’s earlier “letter” as a source of inspiration. Not once does he indicate that his “letter” is strikingly similar to More’s. Yet almost no one cites More; instead, the citation goes to Bostrom.
6. Astronomical Waste
One of the foundational papers of longtermism — the claim that we should become posthuman, colonize space, and flood the universe with digital beings living in giant computer simulations — is Bostrom’s 2003 paper “Astronomical Waste: The Opportunity Cost of Delayed Technological Development.” In 2022, Elon Musk even retweeted a link to this paper along with the line (in the original tweet): “Likely the most important paper ever written.”
This paper played a huge role in my conversion to the longtermist/TESCREAL worldview. But, once again, I was later shocked to discover that the central idea wasn’t Bostrom’s.
In brief, the paper argues that if we colonize space — especially as digital posthumans — the future population could be enormous. Bostrom estimates some 10^38 digital people within the Virgo Supercluster, a cluster of galaxy clusters that includes our own Milky Way. (That’s a 10 followed by 38 zeros, by the way.)
If one accepts a utilitarian perspective, this means the expected value of ensuring the realization of all those future people (assuming they have net-positive lives) far exceeds the value of, e.g., helping poor people today. If you’re a truly altruistic person who aims to maximize your philanthropic impact, then you should focus not on present-day people but those living perhaps millions or billions of years from now. It’s not that future digital people are inherently more valuable. It’s just a numbers-game: there are so many more of them (potentially) than present-day people, they overwhelm the simplistic equations of utilitarianism’s moral mathematics.
Bostrom’s conclusion is that we should colonize the universe as soon as possible, because every single century, decade, and even second of delaying this results in an “astronomical” loss of potential value — i.e., the non-existence of people who could have otherwise been created in some distant galaxy if only we had started colonizing just a bit sooner. However, Bostrom also concludes that “priority number one, two, three and four should … be to reduce existential risk.” Space colonization should therefore be priority number five once we secure our survival. Perhaps you can see why Musk loves this paper.
In the second paragraph of Bostrom’s paper, he writes that “one recent paper speculates, using loose theoretical considerations based on the rate of increase of entropy, that the loss of potential human lives in our own galactic supercluster is at least ~10^46 per century of delayed colonization.” This cites a paper by Bostrom’s colleague, Milan Ćirković, titled “Cosmological Forecast and its Practical Significance,” published one year before “Astronomical Waste.” Curious, I read Ćirković’s piece and was quite surprised to find that he makes an almost identical argument to that of Bostrom, though the details and framing are different.11 At the risk of going into too much detail, here’s what I write about this paper in my 2024 book:
Ćirković argued that physical eschatology paired with recent developments in anthropics (from both [John] Leslie and Bostrom) carry potentially urgent practical implications for “intelligent observers interested in self-preservation and achieving [the] maximum of its creative potential.” Specifically, the timing of space colonization could make a significant difference to the amount of resources available to advanced civilizations, given our best current understanding of our “cosmological situation.”
That’s exactly what Bostrom argues. Even more:
As [Ćirković] expressed the idea, decision-making performed today, as far as humanity is concerned, may have enormous consequences on very long timescales. In particular, an overly conservative approach to space colonization and technologization, may result (and in fact might have already resulted) in the loss of [a] substantial fraction of all possible observer-moments humanity could have had achieved. The term “observer-moment” comes from Bostrom, who defined it as “a brief time-segment of an observer.” Thinking in terms of observer-moments instead of observers is relevant to anthropics — that is, to reasoning about one’s location in space and time based solely on the fact that one is an intelligent observer. But it may also be relevant to estimating our “potential” over time: since “different observers may live differently long lives, be awake different amounts [of] time, … etc.,” counting observer-moments rather than observers can give a more accurate measure of the bigness or value of the future. There could, after all, be a large number of observer-moments even if there are relatively few observers, and vice versa.
The point is that Ćirković used this concept to argue that we should develop advanced technologies and colonize the universe as soon as possible, given that every passing moment our cosmic endowment of negentropy is going to waste as stars burn up their limited reservoirs of hydrogen. Consequently, by failing to advance technology and colonize space, we could lose — and maybe have already lost — a “substantial fraction of all possible observer-moments [that] humanity could have … achieved.” To underline the time-sensitivity of this situation, Ćirković calculated that if a “future hypercivilization” could extract all the energy output of the stars populating the Virgo Supercluster, then “the number of potentially viable human lifetimes lost per century of postponing … the onset of galactic colonization is … 5 × ~10^46.” Thus, assuming some correlation between the number of lifetimes or observer-moments and the fulfillment of our “creative potential,” this provides a very strong prima facie reason against “an overly conservative approach to space colonization and technologization.” (Italics and a paragraph break were added to the original text.)
What’s different between the central arguments and conclusions of Bostrom and Ćirković? Not a ton. Both calculate the costs of postponing space colonization. Both provide estimates of how many people there could be in the Virgo Supercluster. Both argue that we should therefore accelerate the development of technology and colonize space as soon as possible. Bostrom says that the loss of value with every century of delayed colonization “boggles the mind,” while Ćirković writes in similar language that the loss “per a century delay in starting the colonization is astonishing by any standard.”
The biggest difference is that Bostrom frames the issue in terms of “existential risk” (a word that Ćirković doesn’t use) and provides a more explicit ethical foundation — utilitarianism — to support the conclusion he shares with Ćirković. For years, I was under the impression that Bostrom was doing a lot more in his paper than recapitulating Ćirković’s idea of calculating how many future people there could be and then arguing that we should colonize space ASAP to prevent “astronomical waste.”
7. Paperclip Maximizer
If you look up “paperclip maximizer” on Wikipedia, you’ll find a section that opens with this: “The paperclip maximizer is a thought experiment described by Swedish philosopher Nick Bostrom in 2003.” But this is misleading, as it suggests that Bostrom is its progenitor. Indeed, while the idea is regularly attributed to him, it appears that Yudkowsky likely first came up with it.
In a 2018 interview, Sam Harris asks Yudkowsky: “Does this thought experiment originate with Bostrom, or did he take it from somebody else?” Yudkowsky replies: “As far as I know, it’s me.” He adds that it might still be Bostrom’s idea, though he was “the first one to say ‘paperclips’” if the idea originated on mailing lists that he was a part of. (No one, to my knowledge, has double-checked this.)
It’s fine that Yudkowsky doesn’t much care about attribution, but contrast his openness about this issue to Bostrom’s. I have never once seen Bostrom suggest in any interview that he might not have invented the thought experiment. (Let me know if I’m missing something.) He never seems to skip an opportunity to take credit for an idea attributed to him, whether or not that idea is actually his.
Similarly:
8. Pascal’s Mugging
This refers to a thought experiment that shows how maximizing expected value (or utility) — which some would say is integral to rationality — can lead one to make decisions that appear irrational. Expected value is (roughly speaking) what you get when you multiply the probability of an outcome by its value. For example, an outcome with a value of 10 and a probability of .5 has the same expected value as an outcome with a value of 1,000 but a probability of 0.005. (That’s because 10 x .5 = 5, and 1,000 x 0.005 = 5.)
Pascal’s mugging results when enormous values are paired with tiny probabilities. Mitigating speculative existential risks provides an example: if there could be 10^58 happy digital people in the future, as Bostrom calculates, then it might be worth spending huge amounts of money mitigating a highly speculative existential risk scenario even if one judges the probability of that scenario to be tiny. That is to say, multiplying the payoff (value) of the outcome (10^58 happy people) by the (low) probability could still yield a very high expected value.
In 2009, Bostrom published a paper on Pascal’s Mugging in the prestigious philosophy journal Analysis. But the idea was first introduced by Yudkowsky in a 2007 blog post titled “Pascal’s Mugging.” Bostrom briefly gestures in the direction of this fact in a footnote, saying: “Related scenarios have recently been discussed informally among various people. Eliezer Yudkowsky named the problem ‘Pascal’s mugging’ in a post on the Overcoming Bias blog.” Yet it was Bostrom’s name on the Analysis paper, which he was then able to add to his already “impressive” CV, thus boosting his academic “credibility.”
In Yudkowsky’s version, a mugger says to their intended victim: “Give me five dollars, or I’ll use my magic powers from outside the Matrix to run a Turing machine that simulates and kills [an unfathomably large number of] people” (italics added). Even if one deems the mugger’s claim to be extremely improbable, this low probability is offset by the hugeness of the number of people who would be killed. Hence, expected value theory implies that one should hand over the money.
In Bostrom’s version, the mugger says: “I have magical powers. I can give you any finite amount of money that you might ask for tonight. What’s more, I can give you any finite amount of Utility that I choose to promise you tonight” (italics added). Rather than threatening to simulate and kill people, the mugger offers to give the victim a phenomenal amount of money tomorrow if they hand over their wallet right now.
It’s fairly well known that Yudkowsky originally devised the scenario, and that Bostrom basically just re-worded and published it a highly reputable philosophy journal. However, I’d consider it rather odd — and problematic in an ethical sense — if a friend of mine wrote a blog post about an idea, and then I took that idea, changed some minor details, and managed to get a major publication out of it. What do you think?
A Project of Self-Aggrandizement
There are many more examples that could be adduced.12 The point isn’t to shame Bostrom for drawing from other scholars. Every academic does this, of course — it’s the way intellectual progress proceeds. I stand on the shoulders of the giants who came before me, and the next generation will stand on my shoulders by building upon whatever newish ideas (if any) I might have been lucky enough to stumble upon in my brief time here on Earth.
However, in most cases, academics are pretty good at properly attributing the origins of ideas they further develop or elaborate. I do not see this in Bostrom’s works at all. My sense, from reading him since ~2005, is that he goes out of his way to hide the fact that many of his “best” ideas originated in the work of his colleagues. Nowhere in his writings does he say things like the following:
“Reproducing much of John Leslie’s taxonomy of extinction hazards, I will enumerate various scenarios that give rise to what I call ‘existential risk.’”
“As Robin Hanson alluded to, it would be very bad if we were to discover life elsewhere in the universe, given the Great Filter framework.”
“Drawing heavily from the unpublished work of Eliezer Yudkowsky, I will outline an argument for why ASI could bring about the annihilation of humanity.”
“Developing an idea introduced by Stephen Omohundro, I will examine what I call ‘instrumental goals’ that ASI will naturally pursue.”
“Echoing Max More, this ‘letter’ outlines various projects we should pursue in order to become posthuman.”
“I will argue, as Milan Ćirković does, that every century of delayed space colonization results in an astronomical loss of human potential/value.”
Etc.
The vast majority of scholars would have included such remarks if they had written Bostrom’s papers. It’s standard to not take credit for other people’s intellectual labor; it’s common to be generous in citing others. (Plenty of my own scholarship draws from other people, sometimes heavily. But when I do this, I make it clear that I’m relying on their insights. Nothing special here — precisely because it’s standard practice!)
Yes, Bostrom does cite the people above, but not in a way that properly conveys the extent to which his work draws from them — or so I would argue. Pair this with the large volume of evidence that Bostrom thinks of himself as a super-genius of almost messianic significance — he’s saving not just the world, but the entire future of posthumanity — and it starts to look like these weren’t exactly innocent mistakes.
From my perspective, Bostrom is at best a second-rate thinker whose real talent is repackaging other people’s insights in accessible-yet-academic language that appeals to a wide audience. This is what he’s good at — and we should give him credit for that. As Goertzel notes, the guy “is a very good writer, mixing academic rigor with occasional color and humor in a classic European style.”13 But for originality, creativity, and insight, Bostrom gets a D- in my gradebook.
One can see this — or at least I do — as part of a life-long project of self-aggrandizement. And it’s been largely successful: Bostrom has become “one of the most-cited philosophers in the world,” as he boasts on his website. This is, however, thanks to the success of publications like those mentioned above: Superintelligence (12,152 citations on Google Scholar), “Existential Risk” (1,392 citations), “The Superintelligent Will” (674 citations), “Astronomical Waste” (402 citations), “Letter from Utopia” (219 citations), “Pascal’s Mugging” (104 citations), and so on. On his website, Bostrom adds that his
writings have been translated into more than 30 languages; he is a repeat main-stage TED speaker; he has been on Foreign Policy’s Top 100 Global Thinkers list twice and was included in Prospect’s World Thinkers list, the youngest person in the top 15.
This all looks rather less impressive when you realize that the reason he has so many “accomplishments” is because of publications that drew heavily, often without sufficient attribution (in my opinion), from others.
***
While writing this penultimate section, I decided to look into Bostrom’s boast that he’s among the most-cited contemporary philosophers. When Gemini answered my Google query about this, it told me the following:
The claim that Nick Bostrom was the “most-cited professional philosopher under the age of 50” comes directly from his official professional biography used across academic institutions, speaker bureaus, and publishers.14
Oh, so the biography that he wrote is the citation for a claim in his biography! After more digging, I found this ranking from after 2020, which lists him as the 58th most-cited philosopher at the time. Two thoughts:
Does 58 really count as “one of the most-cited philosophers in the world”? I’d have thought this would be reserved for the top 10, or perhaps the top 50. Bostrom is neither in the top 10 nor top 50.
I have never once seen any other philosopher above Bostrom on the list bragging about their ranking. (Have you?) Consider that David Chalmers — who is lionized within philosophy — is listed as #10 in the rankings. Yet nowhere on his website does he mention this. (According to Google Scholar, Chalmers has nearly twice as many citations as Bostrom.) The same goes for Timothy Williamson at Oxford, listed as #8. Even Martha Nussbaum, who’s the most-cited living philosopher, says nothing about it on her website or CV. Why? Because, while academics are expected to proudly list their awards and publications, overt self-promotion is often seen as bad form. It’s cringe-inducing and doesn’t reflect well on the humility of the individual.
But Bostrom, once again, never misses an opportunity to exaggerate and promote himself. It’s what his life-long project of self-aggrandizement is all about.
Why Has No One Said Anything?
An interesting question is why the people from whom Bostrom has borrowed so heavily without proper attribution have been okay with this. Literally no one, to my knowledge, has complained about intellectual plagiarism. Why not? I brought this up to a former researcher at Bostrom’s Future of Humanity Institute (FHI), who responded as follows:
My view, for what it’s worth, is that Bostrom has played fast and loose with the ideas of people who were happy to be plagiarized because they thought “he’s going to be a big Oxford philosopher and having someone like that in our pocket will be good for us and the things we care about so let’s all just go along with it.” I.e. I think maybe you [Émile] are not being hard enough on the people who have let Bostrom plagiarize them without protest, because they only did it because it was in their interests to do so, they weren’t just being charitable to a sub-genius with an ego trip!
On this view, personal self-interest is why no one has spoken out. That seems plausible to me. However, the fact that others are okay with Bostrom taking their ideas doesn’t make it right. Consider a friend of mine who writes an excellent philosophy paper and then says to me (in some parallel universe): “I know you need another publication for tenure, so feel free to put your name on this one and publish it as yours. We’ll never mention this to anyone.” If I were to publish their paper as mine, I would still be committing academic fraud — even if I had their full permission. Standards don’t dissolve if consent is granted.
So, it’s a bit frustrating that no one has said anything. If the people above were to publicly complain, then (a) they’d have a very good case for being upset, and (b) the result could well be a significant scandal for Bostrom. As far as I know, this article is the first to lay out the evidence and make that case. Perhaps it will inspire someone to speak out, especially now that Bostrom is no longer at Oxford.15 We’ll see.
Have you come across important examples that I’ve missed? Please do share them below (or feel free to email me). As always:
Thanks for reading and I’ll see you on the other side!
Simon Knutsson, the author, writes:
I have a bachelor’s and two master’s degrees from the same university in Gothenburg, Sweden, and I have lived almost all of my life in Sweden. I have never heard of any national record related to undergraduate performance. I mentioned his statement “Undergraduate performance set national record in Sweden” to a few people in Sweden who are familiar with the university system in Sweden. One person laughed out loud, another found his claim amusing, and a third found it weird. I called the University of Gothenburg on Oct. 21, 2019, but the person there was not aware of any such records. On Oct. 21, 2019, I wrote Bostrom and asked him about his record. On Oct. 23, 2019, he replied and gave me permission to share his reply in public, the relevant part of which reads as follows:
The record in question refers to the number of courses simultaneously pursued at one point during my undergraduate studies, which – if memory serves, which it might not since it is more than 25 years ago – was the equivalent of about three and a half programs of full time study, I think 74 “study points.” (I also studied briefly at Umea Univ during the same two-year period I was enrolled in Gothenburg.) The basis for thinking this might be a record is simply that at the time I asked around in some circles of other ambitious students, and the next highest course load anybody had heard of was sufficiently lower than what I was taking that I thought statistically it looked like it was likely a record.
Bostrom has repeatedly and conspicuously boasted that he set a “national record,” including on his official CV, and this is what he bases it on? What an arrogant bullshitter!
Note that I am quoting some of these. Furthermore, the order that Bostrom presents them might not correspond to the order in which I’m presenting them, and I have — for the purposes of highlighting similarities — combined a few risks into the same category, e.g., deliberate misuse of nanotech and nanotech accidents are distinct categories in Bostrom’s paper, though I’ve listed them together here.
Note that there is no such thing as a “supervolcano,” despite what Wikipedia says. There are, however, some volcanoes capable of producing supereruptions. I was corrected by one of the leading volcanologists in the world after he read portions of my 2024 book!
Footnote 2 applies to this list from Leslie, too.
I am here combining 6 and 7 in Leslie’s list of “Group 2: Man-made disasters.”
I have not included every potential threat that Leslie specifies. Check out the relevant chapter in his book to compare.
Bostrom and Leslie even approach the topic from a similar utilitarian perspective (Leslie is an “ideal” utilitarian), though this perspective is more clearly evident in Bostrom’s 2003 paper “Astronomical Waste,” discussed below.
Along similar lines, Omohundro also identifies the following as an AI drive: “AIs will try to prevent counterfeit utility” (e.g., they will avoid wireheading). Overall, he enumerates six drives in total, though on my reading these overlap in various ways.
I am here combining #1 and #2 in Omohundro’s list of AI drives.
EDIT (26/7/2026): Note that Ćirković thanks Bostrom, among other people, in the Acknowledgments section of his paper: “The manuscript enormously benefited from discussions with Nick Bostrom, Petar Grujić and Fred C. Adams.” But he does not say that he and Bostrom came up with the idea together, only that his paper benefitted from chats with Bostrom, as well as the physicist Petar Grujić and astrophysicist Fred Adams.
I’ve said nothing about his papers on transhumanism and human enhancement, for example. Many of these draw (heavily) from other transhumanists like Max More, though without making this sufficiently explicit.
Furthermore, I’ve been struck by how similar Bostrom’s “simulation argument” is to versions of the Boltzmann brain problem, the latter of which has been around for quite a while and gained attention shortly before Bostrom published his “Simulation Argument” paper. Bostrom reasons that if our posthuman descendants decide to run a large number of simulations of their ancestors — us — then the total number of simulated people may vastly outnumber the total number of real, actual people living in “base reality” (i.e., non-simulated reality). If you then apply the Principle of Indifference to this, you must conclude that you are far more likely to be in a computer simulation than in base reality.
The Boltzmann brain problem is this: if the universe lasts infinitely (or at least extremely) long, then over the course of endless eons there will be an infinite number of human brains that arise from random thermodynamic fluctuations. These brains will have fully intact memories and perceptions, and the exact same subjective experiences that you have, right now. Consequently, the total number of such “Boltzmann brains” will vastly exceed the total number of real brains. You can then apply the same logic as above to get the conclusion that you’re far more likely to be a Boltzmann brain at the moment than a real brain.
Bostrom claims to have studied theoretical physics, so presumably he was aware of the Boltzmann brain phenomenon. I’m curious whether this is what led him to apply the same reasoning to the issue of computer simulations, despite him not mentioning it. At this point, my article has, I think, clearly demonstrated that Bostrom isn’t in the habit of citing others, because doing so would make his ideas appear less original. I don’t know if the simulation argument is an example of reformulating an idea within a different context (which itself is novel), but I do wonder.
Incidentally, unlike the Boltzmann brain argument, Bostrom’s simulation argument doesn’t work. He writes that “the basic idea of this paper can be expressed roughly as follows: If there were a substantial chance that our civilization will ever get to the posthuman stage and run many ancestor-simulations, then how come you are not living in such a simulation?” The obvious problem is that if our civilization were to “advance” to a posthuman stage and run a huge number of simulations, the Principle of Indifference wouldn’t apply to us. Why? Because we’d know that we’re on the same level of reality as our posthuman successors, and if we’re on the same level, they can’t be simulating us. The Principle of Indifference applies only when you have no additional information for determining which of several possibilities applies. But in this case, we would have information about which level of reality we’re on — namely, the same level as the posthumans running a bunch of simulations. That means, once again, that we can’t be in one of their simulations. I’m honestly rather shocked that Bostrom’s paper made it through peer review!
Although some of Bostrom’s writing, to be blunt, is terrible. For example, his “Letter from Utopia” is so cringe-inducing to me that I have struggled mightily to read the entire short essay in one go. (This was true even when I was a diehard TESCREAList.) It’s authored by someone who is wildly overconfident in their ability to write beautifully and poetically. It’s the work of a poetaster who doesn’t know he’s a poetaster.
There are many examples (read it for yourself), but here are a few to give you a sense:
What is Suffering in Utopia? Suffering is the salt trace left on the cheeks of the oldtimers who were around before.
What is Tragedy in Utopia? It is tragedy enough when Mr. Snowman melts in the spring. …
What is Death in Utopia? Death is the darkness that ultimately surrounds all life.
What is Guilt in Utopia? Guilt is our knowledge that we could have created Utopia sooner.
Oh god help me.
It adds: “Because citation metrics by age are not tracked by a single official body, this description is an aggregated data point originating from Google Scholar rankings for analytic philosophers.” (I’ll compare Bostrom’s Google Scholar citations to David Chalmers’ in a moment.) Here’s a screenshot:
Hmm.
Due in part to a racist email of his that I uncovered in late 2022.
I'm no fan of Bostrom and his adjacent thinkers but certainly reading Superintelligence I thought it was at least implicit that this was one of those "philosophers" who largely curates and summarises the ideas of others. Which is not necessarily a useless endeavour. I've had to read enough "philosophy of neuroscience" theses from the late 90s/early 2000s to see the same pattern of "summarise some introductory neuroscience textbooks in chapters 1-3, summarise some basic philosophy of mind in chapters 4-6, job done."
Thanks for this Émile- I am finishing my book The Case Against The Stars, which internally critiques Bostrom’s case for space colonisation, and learning of his influences is helpful for me to cite them properly.