Existential Risks in an Age of Science

One of the oddities of our age is that there are two apocalyptic sensibilities that have captured two almost completely distinct groups of the scientifically literate not least among the youth: one can be found among those concerned with the effects of global warming produced by human (technological) activity. Scientifically literate apocalypticism is what concerns me in this post. The other can be found by people concerned about and often working in the development of artificial intelligence above all among those shaped by ideas derived from the philosophers associated with Long-termism. I used ‘sensibilities’ in the first sentence because when you try to look more closely at the wide diversity of views within each of the two ‘doomerist’ groups, one finds much heterogeneity in expectations (and the risks/scenarios attached to them). I find the lack of genuine interaction among these two groups very peculiar, but why this is so I will set aside.

I suspect the relative absence of overlap among the two doomerist orientations has allowed the environmental orientation to have become coded Left-centrist, and the other coded Right-Libertarian in many societies. But I don’t think that’s fully solidified, and it’s not my interest today. Of course, many environmentalists tend to treat the growth of datacenters as contributors to the problem they are focusing on.

Scientifically literate apocalypticism is not wholly unprecedented. First, at one point, in living memory of some among us, there was a similar sensibility about the role of nuclear technology, especially atomic weapons, including among many of the people involved in research and development of adjoining technologies. In fact, there existed military/political strategies, based on then cutting-edge formal modeling (and computing), that required from participants in super-power conflict that one credibly aimed at mutual destruction. It is not unlikely one can find more examples among those who were allowed to be aware of the development of weaponized pathogens during the cold war. That is to say, once post Baconian science (and adjoining technologies) progressed sufficiently to conceive of or build weapons of truly mass destruction at limited cost, such apocalyptic sensibilities are to be expected given only minimal assumptions about human nature as we find it.

Second, scientifically grounded apocalypticism is not exclusively inherent in post-Baconian modernity. It’s pretty clear if you read Seneca’s work on comets, Book 7 of Natural Questions, that the planetary science of the ancient Stoics assumed that our solar system was not stable and that catastrophic ends of worlds were a recurring feature of the cosmos. This fit in a wider understanding of the cyclical nature of history. I put all of that slightly anachronistically, but much less than you would imagine. Anyone who looked at the maps of the universe with Descartes’ giant, unstable planetary vortices could contemplate the eternal return of the Stoics.

And familiarity with the discussion in ancient Stoicism through Seneca is almost certainly why one of the first foundational open questions after the publication of the Principia was the stability of the solar system.1 (Attempting to solve the technical problems associated with this drove much of the advances in mathematics, probability theory, and physical astronomy in the eighteenth century.) Long before the Big Crunch and Big Freeze, there were genuine concerns about planetary collisions (evidence of the plausibility of which also piled up as our telescopes got better at checking the surface of the Moon).

Ordinarily I would emphasize the many millenarian-Christian ideas (typically with a sectarian Protestant inflection) that resurfaced in a more secularized register within the Environmental Movement and Long-termism. But as should be clear from my discussion of Stoicism, I deny that ‘scientific apocalypticism’ is necessarily tied to the influence of Christianity, even if it’s pretty clear that many elements in anthropogenic scientific doomerism manifestly are. That we are the cause of our own dread is a by-product of Baconian science, and perhaps something about the ways we conceive of agency since.

Now, the sample size of cases I have sketched is pretty small. Even so, it allows for some fruitful distinctions and observations. For example, eighteenth century doomerism was scientifically productive: scientific focus on it generated lots of new techniques and theories as well as a happy result: the solar system is more stable than we might have hoped. This dissipated the dread at least provisionally. And it probably even lengthened the optimism and providentialism associated with scientific progress. The Big Bang and the Big Freeze re-opened it, but at scales that leave most unmoved; here our cognitive biases prevent acute dread. Lurking in my presentation is the thought that, if we let ourselves be scientifically informed, a process’s timescale and the agent’s window of effective monitoring action matter greatly to our affective stance. Structurally this is analogous to the significance of what I call (recall) ‘Humphreys opacity’ (which I define below).

I wouldn’t know for sure if mutually assured destruction helped prevent nuclear war among super-powers, but the dread of it is much less than it once was. (To what degree fears of it are likely to return as the global system enters an age of mercantile competition is, alas, a tricky question.) This is an interesting case because what unites the two contemporary doomerist groups is that they are also anthropogenic. And it would be nice if ‘governance’ could explain the control and diminution of atomic doomerism.

Now, I hope it is fair to say that what prevents the dissipation of dread about Anthropocene extinction related to climate change is a thoroughgoing disagreement over who may pay to prevent it and far-reaching disagreement over what kind of social world we and our descendants ought to inhabit and how much risk we are willing to let ourselves and others tolerate along the way. Of course, there are other ways of describing such thoroughgoing disagreement. And, if you are a Marxist, and want to blame ‘Capitalism,’ that’s fine by me.

And I hope it is also fair to say that what prevents the dissipation of dread about AGI is that it seems we are entering the age where the machines that may be the source of human extinction will be built by systems of artificial intelligence itself, that is recursive self-improvement. Thinking about recursive self-improvement is not new conceptually. (As my regular readers know (recall here), Paley articulated it as a reductio of atheism.) What is new is that recursive self-improvement seems very likely. And, crucially, because modern artificial intelligence is characterized by various forms of Humphreys opacity we humans would not be able to catch the threat AI forms in real time.2 This kind of opacity (recall) involves the inability to surveil the steps of a process from a known input to a known desirable (or truthful, useful, beautiful, etc.) output in a timely manner to the decision-maker or responsible agent.

What the two contemporary doomerist cases I am focusing on have in common is that no likely discoverable set of facts could dissipate the dread. Although it would be very nice if there were grounds to think the timeframes involved could be lengthened greatly. (Some of the people who focus on the existential risk of AGI are techno-optimists about climate change.) Unfortunately, there are powerful currents in both cases that strongly suggest events are accelerating toward doom. It seems that in both cases there is a converging consensus that the root cause of the problem prevents an endogenous solution; political life has to intervene, but how it should do so is very contested. (Again, if you are a Marxist you are allowed to claim the ultimate root of the problem is ‘capitalism’ or ‘neoliberalism.’)

Now it is somewhat peculiar — and this is why the distinctness between the two groups matters — that the way AI induced existential risk is formulated and debated among those who take it very seriously presupposes the very thing, ‘alignment,’ whose absence is one of the main causes for why the climate activists have failed so far. Here a pun is masquerading as an argument. Of course, ‘alignment’ itself covers many different kinds of phenomena, and you may claim that these keep the two doomerist orientations distinct.

Part of what’s at stake here is the thought that even if AI is one thing/system (which I would argue it is not) what it means to make it reliably and minimally cohere with ‘our’ aspirations ultimately bottoms out in the very political (and other) disagreements the climate case has manifested. That’s to be defended in future posts.3 (My regular readers know I am interested in the ways political conflict can be turned into a species of discovery and governance, and the many ways attempts at tackling Humphreys opacity can generate higher forms of it.) But for now: the apparent lack of curiosity about the very real political failure(s) of the climate movement among those that non-cynically focus on the existential risk of recursive self-improvement is one of the main reasons why I wouldn’t be too optimistic that the dangers of recursive self-improvement will be managed well. (Again, Marxists can chant ‘Capitalism.’)

That’s the main point. But here’s a closing thought. You might have suspected (or feared) that my invocation of Seneca would have ended up in preaching equanimity, amor fati, or suicidal despair. But as Adam Smith (here) noted, Stoicism could drive people into political activism. For Stoicism’s “general tendency was to animate them to actions of the most heroic magnanimity and most extensive benevolence.” If the doomerists are right, then how to merge successfully apocalyptic sensibilities with effective political philanthropy will be our era’s defining challenge.