This is a cross-post from my blog post.

Over the past decade, the effective altruism movement has become increasingly focused on “existential risk,” the idea that, this century, there’s a significant chance that humanity will go extinct, become permanently disempowered, or otherwise lose almost all of its future potential.

In this post, I’m not going to argue that, given what we know, we should believe that existential risk is high or low. Instead, what I’m going to argue is that existential risk is an extraordinary claim and that, as such, we must have extraordinary evidence to be concerned about it. Whether we have such evidence I’ll leave to the reader.

My most general argument for this is simply that existential risk represents a dramatic state change in the world and that, the more dramatic of a state change someone is arguing for, the more evidence they must supply to convince us of it. If your friend tells you that one of your windows could get broken this year because of the upcoming hail storm, you have good reason to believe them. But, if they tell you that every single house in your entire state will be completely destroyed, they need to provide extreme evidence to substantiate such a claim.

Extinction

The dominant existential risk that effective altruists worry about is human extinction.

But, notably, in order for this to happen, there has to be an extremely specific set of conditions. It can’t merely be that some event, say a nuclear war or a pandemic, causes a large number of people to die. Instead, it has to be that, for some reason, every single last human is killed. The only mechanisms I can really think of that would cause this is either that some entity actively seeks out and kills every human or that every part of the Earth becomes uninhabitable for a significant duration of time.

One way some argue this could occur is that humanity will develop a superintelligent AI that will systematically seek out and kill every last human. But, to believe that this could happen, we have to believe the claims that humanity can make such an intelligence, that humanity will make such an intelligence, and that, if we do, it will seek to destroy us.

Another way some argue that this could occur is that humanity will develop engineered viruses that infect and then kill everyone. But, to believe that this could happen, we need to have good evidence to think that such a virus (one that spreads rapidly and has 100% fatality) can be engineered, that people who are unreachable (because they’re in bunkers or outer space or Antartica) will also die, and that humanity will be unable to create a cure for such a virus in time.

The last way that some argue this could occur is that two great powers will engage in a nuclear war that causes a nuclear winter. This could cause crop failures around the world, and billions would certainly die. But, it’s important to remember that nuclear winters are not expected to last more than a decade. Given this, we need to believe not only that total war between nuclear powers will occur and that this war will result in a nuclear winter, but also that those living underground or in protected areas will also perish before the surface of the Earth becomes capable of agriculture again.

Some may take the claims of a few experts or their own “vibes” as evidence that we should be very worried about existential risk, but it’s important to remember that humans have a natural proclivity towards believing that their generation could be the last. (According to Pew, “39% of adults believe that humanity is ‘living in the end times.’”) Given this, we can’t merely rely on the beliefs of a few experts or our personal opinions. Instead, we need the concern of a significant portion of experts to really justify concern. Some may believe they have strong enough evidence to worry, but it’s important to remember the mountain of learning that has to be done to be at all confident about these things. (For instance, it seems pretty hard to say whether extremely deadly viruses could be engineered without knowing a lot about virology.)

Disempowerment

Another existential risk effective altruists worry about is disempowerment.

The basic concern is that, if superintelligence is developed, it actually may not eliminate humanity because that would be a waste of resources. Instead, it would simply disempower us by eliminating our ability to make choices about the future of our species. Whether this is more or less likely than the extinction scenario I’ll leave to the reader.

Irreversible Civilizational Collapse

The last existential risk effective altruists worry about is irreversible civilizational collapse.

Some effective altruists believe that, even if the risks in the extinction section wouldn’t actually kill everyone, they would still kill enough people that humanity would be unable to re-industrialize afterwards, possibly because it has gone through all of the easily accessible fossil fuels on Earth. (The thought is that, if you can’t get coal, you might not be able to make a steam engine and then not be able to make almost anything else.)

I think this is also an extraordinary claim because it supposes that there’s a significant risk of almost everyone dying, that, if almost everyone died, civilization would become set back to pre-industrial development, and that, if this were to occur, humans would never be able to re-industrialize despite having vast amounts of knowledge, technology, and creativity.