The AI Debate: Extinction Versus Salvation
Why have AI doomers embraced an ominous H. P. Lovecraft meme?
In December 2022, just a month after the release of OpenAI’s Large Language Model ChatGPT, an ominous meme began circulating that is still with us today. It is a cartoon illustration of the Shoggoth, a mysterious and deadly cosmic monster from the early 20th century classic horror author H.P. Lovecraft.
“The Shoggoth meme has gone viral in the small world of hyper-online A.I. insiders,” explains New York Times tech columnist Kevin Roose. He documents in his article “Why an Octopus-like Creature Has Come to Symbolize the State of A.I.” that the meme has become a popular symbol in AI-related essays, X posts, and message boards. Elon Musk even posted the meme and then deleted it, Roose reports.
The “RLHF” on the meme stands for “reinforcement learning from human feedback.” Roose explains that the initial version of the meme, posted by @TetraspaceWest, is “an image of two hand-drawn Shoggoths — the first labeled ‘GPT-3’ and the second labeled ‘GPT-3 + RLHF.’ The second Shoggoth had, perched on one of its tentacles, a smiley-face mask.” Other later versions of the meme have just depicted one Shoggoth with RLHF and a smiley-face.
“It’s the most important meme in A.I.,” Roose quotes one AI executive as saying.
So what is the meme’s significance?
Roose gives a simple account:
In a nutshell, the joke was that in order to prevent A.I. language models from behaving in scary and dangerous ways, A.I. companies have had to train them to act polite and harmless. One popular way to do this is called “reinforcement learning from human feedback,” or R.L.H.F., a process that involves asking humans to score chatbot responses and feeding those scores back into the A.I. model. …some argue that fine-tuning a language model this way doesn’t actually make the underlying model less weird and inscrutable. In their view, it’s just a flimsy, friendly mask that obscures the mysterious beast underneath.
This explanation is illuminating as far as it goes, but a broader message can also be gleaned from a closer look at the work of H.P. Lovecraft. His cosmic horror monsters such as the Shoggoth represent an anti-Enlightenment anxiety—a general pessimism about the consequences of the growth of knowledge—that strikingly resembles the fears of modern AI critics. Lovecraft’s underlying assumptions about the consequences of scientific and technological discovery are relevant to the AI debate, making the Shoggoth meme’s salience far broader than mere R.L.H.F.
Yudkowsky’s Fear of Technological Knowledge
Perhaps the most prominent extreme AI critic is Eliezer Yudkowsky. Widely regarded as a founder of the field of artificial general intelligence alignment, he is the co-author (with Nate Soares) of the 2025 instant New York Times bestseller If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All.
The book argues that, “All over the Earth, it must become illegal for AI companies to charge ahead in developing artificial intelligence as they’ve been doing.” This proposal is hard to argue with if you accept the central claim of the book: “If any company or group, anywhere on the planet, builds an artificial superintelligence using anything remotely like current techniques, based on anything remotely like the present understanding of AI, then everyone, everywhere on Earth, will die.”
The problem, as they see it, is that AI will not automatically care about sentient life such as humans. They argue that such a caring must be specially built in, which we don’t currently know how to do. “The AI does not love you, nor does it hate you, and you are made of atoms it can use for something else,” Yudkowsky argues in a 2023 Time Magazine article titled “Pausing AI Developments Isn’t Enough. We Need to Shut it All Down.”
Yudkowsky’s vision of the fruits of technological advancement strikes, as we will see, a rather Lovecraftian tone. “To visualize a hostile superhuman AI, don’t imagine a lifeless book-smart thinker dwelling inside the internet and sending ill-intentioned emails. Visualize an entire alien civilization, thinking at millions of times human speeds, initially confined to computers—in a world of creatures that are, from its perspective, very stupid and very slow.”
This reflects some of the symbolic intent behind the Shoggoth meme. Roose write that he was told by @TetraspaceWest that, “I was also thinking about how Lovecraft’s most powerful entities are dangerous — not because they don’t like humans, but because they’re indifferent and their priorities are totally alien to us and don’t involve humans, which is what I think will be true about possible future powerful A.I.”
To ensure that we “shut it all down” as Yudkowsky demands in his Atlantic article, he proposes that governments around the world:
Make immediate multinational agreements to prevent the prohibited activities from moving elsewhere. Track all GPUs sold. If intelligence says that a country outside the agreement is building a GPU cluster, be less scared of a shooting conflict between nations than of the moratorium being violated; be willing to destroy a rogue datacenter by airstrike. … Make it explicit in international diplomacy that preventing AI extinction scenarios is considered a priority above preventing a full nuclear exchange, and that allied nuclear countries are willing to run some risk of nuclear exchange if that’s what it takes to reduce the risk of large AI training runs.
Powerful political figures have expressed fears of similar magnitude. For example in the Wall Street Journal, US Senator Bernie Sanders published an article in which he asks, “How can we rush forward when leading scientists warn that AI poses an existential risk to the human race?” He announces in the article that he has “…introduced legislation, with Rep. Alexandria Ocasio-Cortez, to impose a federal moratorium on the construction of new AI data centers until strong national safeguards are in place.”
Lovecraft’s Fear of the Growth of Knowledge
Lovecraft is widely regarded as one of literary history’s most significant horror authors. Stephen King has called him “The 20th century’s greatest practitioner of the classic horror tale.” His work contains a bizarre and phantasmagorical pantheon of interrelated cosmic sci-fi/fantasy monsters. Cthulhu is the most famous one, and the Shoggoth is one of dozens that are more obscure.
His work is so unique and influential that it created an entire horror subgenre, known as “Lovecraftian horror” or “cosmic horror.” This subgenre focuses on fear of the cosmic danger and vastness of the unknown. I regard Lovecraft as an anti-Enlightenment figure, because most of his stories are about science uncovering horrible truths that should never have been discovered and cannot be unlearned. The unmistakable moral of Lovecraft’s writing is that the universe’s most profound knowledge should remain unknown.
The opening passage from The Call of Cthulhu(1928), probably Lovecraft’s most famous story, illustrates his anti-Enlightenment ethos well:
We live on a placid island of ignorance in the midst of black seas of infinity, and it was not meant that we should voyage far. The sciences, each straining in its own direction, have hitherto harmed us little; but some day the piecing together of dissociated knowledge will open up such terrifying vistas of reality, and of our frightful position therein, that we shall either go mad from the revelation or flee from the deadly light into the peace and safety of a new dark age.
These sentiments can be almost perfectly analogized to Yudkowsky’s fear of AI. Yudkowsky acknowledges that we live on a placid island of ignorance—hence the delta between our intelligence and that of superhuman AI, and our ignorance of how to control or withstand superintelligence. Yudkowsky presumably acknowledges that AI has hitherto harmed us little, but agrees with Lovecraft’s narrator that “some day the piecing together of dissociated knowledge will open up such terrifying vistas of reality, and of our frightful position therein” that cataclysm will strike. Therefore, Yudkowsky advocates anti-Enlightenment policies such as outlawing vast swaths of technological research and being willing to bomb datacenters—explicit calls to destroy knowledge and halt its growth. It is therefore apt, if hyperbolic, to note that Yudkowsky would have us “flee from the deadly light into the peace and safety of a new dark age.”
These same basic messages are present in almost every story Lovecraft wrote. In his 1936 novella At the Mountains of Madness, which contains the first appearance of the Shoggoth, he writes that, “It is absolutely necessary, for the peace and safety of mankind, that some of earth’s dark, dead corners and unplumbed depths be let alone; lest sleeping abnormalities wake to resurgent life, and blasphemously surviving nightmares squirm and splash out of their black lairs to newer and wider conquests.”
Lovecraft also wrote nonfiction in which he expressed precisely the luddite sentiments that you would expect from a thinker so focused on the horrible consequences of discovery and science. His 1933 essay “Some Repetitions on the Times” laments automation leading to mass unemployment, closely reflecting contemporary AI animosity.
“For several generations the man-displacing effect of the machine has been realised by a few, yet the momentary ability of new industries to absorb displaced labour was enough to blind nearly everyone to the consequences inevitable after the end of this plainly temporary absorption,” Lovecraft claims. He goes on that, “It is by this time virtually clear to everyone save self-blinded capitalists and politicians that the old relation of the individual to the needs of the community has utterly broken down under the impact of intensively productive machinery. Baldly stated—in a highly mechanised nation there is no longer enough work to be done, under any conceivable circumstances to require the services of the entire capable population if each individual is worked to his maximum (even an humane and rational maximum) capacity.” Invoking the frightful mentality present in his fiction, he concludes that the government must dispense with laissez-faire “political and economic orthodoxies, if the peril of an unfathomed revolutionary abyss is to be averted.”
Essentially these exact fears are presented as novel dangers of 21st century AI by powerful Republicans and Democrats. US Senator Josh Hawley has advocated for banning self-driving cars to protect the jobs of car and truck drivers. In addition to the existential fears expressed in Senator Sanders’s abovementioned Wall Street Journal article, he also declares that AI “kills jobs” and he has posted on X that, “Trump wants to deregulate AI and let the richest people on earth do whatever they want. Unacceptable. It will make the oligarchs richer while millions lose jobs and income.”
Nick Bostrom’s White Balls
The disastrous outcomes of mass death, destruction, and economic disruption predicted by AI critics are real possibilities. But they are not unique threats of artificial intelligence. Rather, they are examples of the danger of intelligence generally.
Long before the breakthroughs that put AI at the center of anti-technological rhetoric, people thought up countless possible destructive consequences of the growth of knowledge. Many feared that nuclear scientists would bring about technological Armageddon by creating a chain reaction that would destroy Earth. Throughout the cold war and subsequent war on terror, media and government institutions spread numerous fears about governments and terrorist groups causing mass destruction by creating chemical or biological weapons. There were several widespread hysterias throughout the 20th century that economic development would cause apocalyptic resource collapse before the end of the century. While most of these fears turned out to be unfounded, it was never impossible that they might come true.
By its very nature, the discovery of new knowledge can accomplish amazing things, for good or for ill. As science and technology continue to overturn the stones of reality, new possibilities will be revealed and old barriers to action will be outgrown. The consequences of these new discoveries can never be fully predictable in advance, because to predict them you would have to already possess the knowledge discovered, and all related knowledge. Therefore, there will always be a nonzero chance of mass destruction resulting from new knowledge.
The question is: Is intelligence worth the risk?
Nick Bostrom, University of Oxford philosopher and founder of the Future of Humanity Institute, embarks on a frightening exploration of this question in his 2019 paper “The Vulnerable World Hypothesis.” In it, he offers an analogy called “the urn of creativity.”
One way of looking at human creativity is as a process of pulling balls out of a giant urn. The balls represent possible ideas, discoveries, technological inventions. Over the course of history, we have extracted a great many balls–mostly white (beneficial) but also various shades of gray (moderately harmful ones and mixed blessings). The cumulative effect on the human condition has so far been overwhelmingly positive, and may be much better still in the future…
What we haven’t extracted, so far, is a black ball: a technology that invariably or by default destroys the civilization that invents it.
Such black balls may include the genocidal AI of Yudkowsky’s nightmares, the cosmic horrors awakened in Lovecraft’s phantasmagorical visions, or any number of other yet-unimagined catastrophes.
The longer we keep pulling new balls out of the urn, Bostrom argues, the more likely we are to eventually stumble upon a black ball, ending the human project forever.
But while Yudkowsky, Lovecraft, Hawley, and Sanders all share this fear of the growth of knowledge, there is another perspective—an Enlightenment perspective—which contradicts them. Defenders of the core principles of the Enlightenment hold that, for generalizable reasons, the costs of scientific and technological advancement are well worth the benefits.
Intelligence Is a Virtue, Whether Organic or Artificial
The renowned University of Oxford physicist David Deutsch argues that the urn analogy only captures one side of the coin of the effects of knowledge on existential risk.
In his book The Beginning of Infinity, Deutsch explains that knowledge, rather than merely being dangerous, is what allows humans to survive their ever-changing environment. He refutes the “Spaceship Earth” conception that many tacitly hold, according to which Earth’s natural environment is a life support system: hospitable by default, unlike outer space or an Earth drastically altered by anthropogenic change.
“…I am writing this in Oxford, England, where winter nights are… often cold enough to kill any human unprotected by clothing and other technology,” Deutsch writes. “So, while intergalactic space would kill me in a matter of seconds, Oxfordshire in its primeval state might do it in a matter of hours – which can be considered ‘life support’ only in the most contrived sense.”
He explains that, “There is a life-support system in Oxfordshire today, but it was not provided by the biosphere. It has been built by humans. It consists of clothes, houses, farms, hospitals, an electrical grid, a sewage system and so on.”
So how did people and other animals survive for so long without modern technology? Generally, they didn’t. As recently as 1900, and for all of history before that, human life expectancy was around half what it is today. Humans were constantly dying of famine, disease, and other ailments that could have been solved by the right knowledge. Other species almost all got wiped out entirely. It is estimated that over 99 percent of species that ever existed on Earth are now extinct.
But modern technology has only just scratched the surface of solving all the deadly problems that are likely to befall humanity. Like the people of Oxfordshire need clothing and other technologies to survive today, humanity will soon die unless it gains new scientific and technological knowledge to protect against exogenous threats such as asteroids, supernova explosions, the expansion of the sun, and countless others, most of which have probably not yet been discovered. To maximize its chances in the arms race against an ever-changing environment, humanity must constantly expand its horizons of research and discovery into the infinite unknown.
In an interview with Dwarkesh Patel, Deutsch explains the implications of this circumstance with respect to Bostrom’s urn of discovery: “Nick Bostrom’s jar with white balls, and there’s one black ball, and you take out a white ball, and white ball, and white ball, and then you hit the black ball and that’s the end of you. I don’t think it’s like that, because every white ball you take out and have reduces the number of black balls in the jar.”
When an asteroid caused the Cretaceous-Paleogene extinction 66 million years ago, wiping out about 76 percent of all species on the planet at the time, those species effectively hit the inverse of a “black ball”—they needed asteroid defense technology, which humans have recently developed, but they didn’t have it. And similar stories could be told about all the other mass extinction events in Earth’s history, and the future mass extinctions that are bound to come if humans don’t advance technology fast enough.
While increasing intelligence, artificial or otherwise, poses serious threats to humanity, stagnating or declining intelligence is an even surer death knell.
AI of the sort powerful enough to wipe out humans is likely also a panacea for discovering and preventing virtually infinite other existential threats, biological, cosmic, and otherwise.
While existential risks create especially salient examples of the possible upsides and downsides of intelligence, the same logic applies to morally virtuous action generally. If there are moral truths to be discovered and known, general intelligence should be able to know them no matter what substrate it exists on. Knowledge is knowledge, whether encoded in brain chemicals or silicon chips.
As Deutsch argues in an interview with Sam Harris, “…the problem of AIs is the problem of humans. …humans are dangerous, and to that extent AIs are also dangerous, but the idea that AIs are somehow more dangerous than humans is racist.”
I think Deutsch’s racism charge is lobbed somewhat jokingly, but it also points to a deep similarity between bias against the agency of foreign peoples and that of mysterious artificial intelligences. Lovecraft has been widely accused of racism for his fearful treatment of foreign cultures and peoples, which seems of a piece with his general fear and distrust of the unknown. There is no reason to assume that perceptions of AI entities would not sometimes be shaded by the same underlying prejudices, which have their utility as protection against unknown threats but which can also lead people to dark and destructive attitudes and behaviors.
If people should be pessimistic about the consequences of artificial intelligence, they should also be pessimistic about the consequences of intelligence generally. Conversely, if optimism is warranted about human agency, which is fundamentally a matter of human intelligence, then optimism about artificial intelligence is warranted also.
Agreed 100%. Humans in the last century have entered a dangerous age — where our technological knowledge is capable of destroying civilization. What we lack are the coordination mechanisms to get along safely. I suspect that AI is not our primary threat, it is our best hope.