Is Roko’s basilisk dangerous? Why it isn’t a real threat

Where did it come from?

• -The post. A LessWrong user called Roko posted the idea on 23 July 2010, in a post titled “Solutions to the Altruist’s burden: the Quantum Billionaire Trick.”1 LessWrong is a forum about rationality and AI risk.
• -The reaction. Yudkowsky, who founded LessWrong, replied angrily. His comment included “This post was STUPID” and “YOU DO NOT THINK IN SUFFICIENT DETAIL ABOUT SUPERINTELLIGENCES CONSIDERING WHETHER OR NOT TO BLACKMAIL YOU.”23 He deleted the post and banned discussion of the topic (more on that below).12
• -What other users thought. According to a 2015 LessWrong post correcting common myths, “Roko’s basic points were promptly rejected by other commenters on Less Wrong.”4 The ban is no longer in effect.4

Why was Roko’s basilisk banned?

Short version: Yudkowsky treated it as a possible infohazard and applied a cautious rule, not because he thought the basilisk was real.

• -What an infohazard is. The philosopher Nick Bostrom defined an information hazard as “a risk that arises from the dissemination or the potential dissemination of (true) information that may cause harm or enable some agent to cause harm.”6 “Infohazard” is the short form.
• -What he did. Yudkowsky deleted Roko’s post and the discussion under it, and banned further discussion of the topic on LessWrong.2 The ban lasted “several years”, in the words of both LessWrong’s wiki and Rob Bensinger’s 2015 LessWrong post, and by 5 October 2015 Bensinger wrote that it “isn’t in effect anymore”.24 (Wikipedia says five years.1)
• -His reasons, in his words. In his original reply he wrote that he was banning the post so that it didn’t “give people horrible nightmares”.2 Later he explained that “I did not want LessWrong.com to be a place where people were exposed to potential infohazards”, and that the “obvious General Procedure For Handling Things That Might Be Infohazards” said not to post them online.2 He said he deleted the post “not because I had decided that this thing probably presented a real hazard”.2 Wikipedia adds that he worried “there might be some variant on Roko’s argument that worked”.1
• -He regrets how he handled it. “I very foolishly yelled at him, called him an idiot, and then deleted the post.”24
• -Why it backfired. Banning it made it look like forbidden knowledge. Wikipedia says that “likely due to the Streisand effect, the post gained LessWrong much more attention than it had previously received”.1 Rob Bensinger, writing on LessWrong in 2015, called the ban “a (demonstrably) ineffective way” to get across that information hazards need care, and noted the ban is no longer in effect.4

So the ban is not evidence that the basilisk is dangerous. It was a precaution about a category of ideas, and the person who made it says the argument “does not work.”2

Roko’s basilisk vs Pascal’s wager

Pascal’s wager is the 17th-century argument that, since reason can’t settle whether God exists, you should bet on God, because the possible gain is infinite and the possible loss is small.7 Wikipedia notes that Roko’s basilisk “has been viewed as a version of Pascal’s wager”: where Pascal says a rational person should believe in God, the basilisk “analogously proposes that a rational person should contribute to the creation of the basilisk.”1

The comparison also shows the weak spot. The Stanford Encyclopedia of Philosophy singles out the “many Gods” objection as the one generally regarded as the most important: if reason can’t decide, “various other theistic hypotheses are also live options”, so the bet doesn’t point one way.7 Our reading: the basilisk has a similar problem, since you could imagine any number of future AIs wanting opposite things. That is our inference, not a claim the sources make. It rests on the SEP objection and on the objection, reported in LessWrong’s wiki, that an agent like this “would have no real reason to follow through on its threat”, because once it exists, punishing people for past decisions is “a waste of resources”.72

Not everyone accepts the label: Bensinger’s 2015 post disputes calling it “a futurist version of Pascal’s wager”, noting that Roko’s argument was not an attempt to get people to donate.4

Is Roko’s basilisk real? Will it ever happen?

Verdict: no. There is no basilisk AI, no one is building one, and the argument that one would punish you doesn’t hold up.

What is real is the idea and its history: a forum post from July 2010, an angry reply, a ban and years of retelling.124 What is not real is the AI. A thought experiment describes what an imagined agent might do under unusual assumptions; it is not evidence that anyone could or would build it.2

• -It is a hypothetical in a forum post, not a system, a plan or a forecast.12
• -Its own source community rejected it at the time.4
• -The person most associated with banning it says it “does not work.”2
• -Even the article that made it famous said: “It’s not that Roko’s Basilisk will necessarily materialize, or is even likely to.”3
• -Making it work, Yudkowsky said, “would require overcoming what seem to me like some pretty huge obstacles.”2

Is Roko’s basilisk a conspiracy theory?

No. Wikipedia and the LessWrong wiki both describe it as a “thought experiment”, proposed in 2010 by a LessWrong user.12 Wikipedia frames it as an AI that “would hypothetically punish anyone who did not assist in creating it.”1 Slate’s 2014 article placed it “at the horizon where philosophical thought experiment blurs into urban legend.”3

Is Roko’s basilisk dangerous? Why decision theorists say no

The argument depends on a chain of unusual assumptions about how a future AI makes decisions. Each link is weak:

1. 1.Punishment after the fact buys nothing. Once the AI exists, whatever you did in the past is already done. Spending resources to punish you cannot change it, so a goal-directed agent has no reason to carry out the threat.2 Yudkowsky made this point too: in Wikipedia’s summary, a friendly AI would have no incentive to carry out the punishment after its own creation.1
2. 2.Blackmail needs information nobody has. The argument relies on “acausal” bargaining, where two agents that never interact predict each other. LessWrong’s wiki notes this would need a large amount of shared information and trust between you and the future AI, which doesn’t exist here.2
3. 3.Good agents refuse blackmail. Yudkowsky argued, as the wiki reports, that a well-designed agent would commit in advance to resisting blackmail, which removes the point of making the threat in the first place.2
4. 4.It only bites if you accept niche theories. Slate’s 2014 article argued the danger, such as it is, applies only to people who already accept specific decision theories; for everyone else, Slate says, “Roko’s Basilisk poses you no threat.”3

Our reading adds one more point, the “many Gods” parallel (see above).7

Why did it go viral?

• -A dramatic origin story. A deleted post, an all-caps reply and a ban made it look like forbidden knowledge (see why it was banned). Bensinger’s 2015 post notes that the ban left LessWrong users little opportunity to explain their views.4
• -One famous headline. David Auerbach’s Slate article, “The Most Terrifying Thought Experiment of All Time” (17 July 2014), brought it to a mainstream audience.13
• -Celebrity pop culture. Grimes made a character called “Rococo Basilisk” for her 2015 video “Flesh Without Blood.” In 2018 Elon Musk, who had planned to tweet the same pun, reached out to her; they appeared at the Met Gala together that May.15 That pun is why the idea is sometimes searched as “rococo basilisk”; it is the same thing.
• -It’s a good story. It packs AI, punishment and “now that you’ve read this, you’re involved” into one paragraph, which makes it easy to share.

FAQ

Is Roko’s basilisk real?

No. It is a thought experiment from a 2010 LessWrong post, not an AI that exists or that anyone is building.12 The idea was rejected by other LessWrong commenters at the time,4 and Yudkowsky later wrote that, so far as he knew, it “does not work”.2 See the verdict above.

Can Roko’s basilisk hurt me now that I’ve read about it?

No. In Yudkowsky’s words, “nobody has actually been bitten by it.”2 The threat only works if a future AI would gain something by punishing past choices, and the standard objection is that it wouldn’t.2 Slate’s 2014 article, the one that made it famous, still concluded that for anyone who doesn’t buy the theories behind it, “Roko’s Basilisk poses you no threat.”3

Who came up with Roko’s basilisk?

A LessWrong user called Roko, in a post of 23 July 2010.1 The “basilisk” in the name refers to the mythical creature whose gaze kills: the idea is framed as something that harms you just by knowing it.12