Hide table of contents

Crosspost from my Substack

1.

When I get into my car, I wear my seatbelt. This is not because I think I will crash. In fact, if I was getting into my car and some smug researcher told me, “Erm, actually, there is a 10% chance you’ll crash”, I would laugh and ignore him. And then I would wear my seatbelt anyways.

To see why, here’s a payoff matrix:

subscribe to help support me!

 

Notice that if I wear my seatbelt and don’t end up crashing, I’ve spent 2 seconds of effort unnecessarily. On the other hand, if I wear my seatbelt and end up crashing, I have saved my life.

There is a saying when you buy a gun: “treat it as if it were always loaded”. This applies even if you are almost certain the gun is not loaded. In fact, if I bought a new gun and some researcher told me “there is a 10% chance your gun is loaded”, I would laugh and ignore him. And then I would still gingerly put the gun away, pointing the muzzle at the ground.

To see why, here’s a payoff matrix:

Notice that if you mistakenly treat the gun as loaded when it is really empty, you waste some energy being a worrywart. But if you mistakenly treat the gun as empty when it is really loaded, you risk the possibility of killing someone.

2.

A few weeks ago I was quite skeptical of AI risk. I read Ramez Naam and Arvind Narayanan and David Thorstad. I think they give good counterarguments against more extreme forms of fast takeoff. I remain uncertain about extinction risk.

At this stage, I don’t think it is possible to be certain.

First, because AI is a genuinely novel technology, without historical precedent. I hope skeptics will at least grant this. Everybody who predicts that capabilities will certainly speed up or certainly slow down is probably bullshitting. I think there are doomers who are also irresponsibly certain.

Second, because the frontier labs have locked their most capable models behind closed doors. Unless you are a lab employee, your lackluster experience using ChatGPT doesn’t count for much.

 

That is to say there is genuine uncertainty. Evaluating AI risk is not like evaluating climate risk or something, where there is empirical evidence that could settle the matter. We are driving blind.

However, just because there is uncertainty does not mean we can retreat to our priors.

I think there is ~0% chance a hurricane hits my home tomorrow. If I opened the news and saw 50 forecasters saying a hurricane would hit, and 50 forecasters saying it wouldn’t, I would have no idea who to believe. But I’d still probably get out of town.

This is the situation with AI.

3.

There are many intelligent, well-researched pieces against AI takeover. However, I am not sure if skeptics are aware of how outnumbered they are. This is not just Jacob Coxon or Eliezer Yudkowsky or a sex-cult in the Bay Area.

Here is a statement signed by Geoffrey Hinton, Yoshua Bengio, Demis Hassabis, Sam Altman, Dario Amodei, Bill Gates, Ilya Sutskever, Igor Babuschkin, Shane Legg, and hundreds of other researchers, professors, and CEOs.

Go read the wording of the statement. Seriously, stop and go read it and then come back.

Here is a survey of 1,580 AI researchers who published papers at top venues (NeurIPS, ICML, etc.) They were asked to put a probability on X-risk.

A full 81% of them think there is at least a 1% chance. The median response says that the chance of extinction or severe disempowerment from AI is 10%.

By the way, this survey probably understates the estimates, since it overrepresents academia instead of employees in frontier labs who are actually working with the most powerful models (that we, by the way, cannot see at all).

Evan Hubinger, Alignment Lead at Anthropic

These are not the opinions of a few forecasters. These are widespread in the field.

I am really worried this topic is getting perceived as momentary hype. Like when people thought Jacob Coxon resigned to boost Anthropic’s market share. Maybe you are receiving it as waves of Twitter hype. Here are some people who have thought about AI risk:

  1. Alan Turing, 1951: “Once the machine thinking method had started, it would not take long to outstrip our feeble powers.”
  2. I.J. Good, 1965: “The first ultra-intelligent machine is the last invention that man need ever make”
  3. Vernor Vinge, 1993: “Shortly after, the human era will be ended.”
  4. Eliezer Yudkowsky, 2008: “The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else.”
  5. Nick Bostrom, 2014: “Before the prospect of an intelligence explosion, we humans are like small children playing with a bomb.”

Also note that both Anthropic and OpenAI were founded explicitly because they were worried about X-risk.

In other words, say what you want about X-risk, but it is not the yearly wave of doomsday paranoia. This has been theorized since before the start of this technology.

Bill Gates, 2026

4.

Here’s where I’m at.

AI capabilities are clearly increasing extremely fast. Although I’m still skeptical of fast takeoff, it is clear this is not a normal technology.

In unprecedented circumstances, you shouldn’t cross your arms and ask for a concrete scenario (spoiler: nobody fucking knows). You shouldn’t cite lack of empirical evidence. (No technology had ever autonomously planned and executed a felony until July of this year). That is what it being “unprecedented” entails.

Most importantly, you shouldn’t use uncertainty as a way to retreat to skepticism.

I also think it is extremely irresponsible to form an opinion based on vibing out meta-level narratives about who is deluded and who is not.1

I was doing this earlier this year: Are the doomers just intelligence-worshipping nerds in a hype-driven bubble? Are the skeptics just unimaginative anthropocentrists in an anti-tech bubble?

At some point, you realize that seeking truth through these meta-level narratives is dumb and fruitless. There are people who know much more than you, who are, at any one point, engaged in a hundred object-level debates that are out of your (and my!) depth. You can’t discard their opinions by forming some folk psychology about them.

I have no certainty at all. I don’t have a p(doom). I have no idea what’s going to happen next year.2

5.

But, I do know this. If things go well, I will live another 60 years. My little brother and sister will live 70.

I am really, really frightened of 5 years of AI progress at the current speed, let alone 70. I am worried for the lives of my friends and family. I am worried we will not get to grow up.

I think if we end up slowing down a few years early, so be it. I really don’t want to be too late.

Here is the payoff matrix for existential risk:

I think we should wear our seatbelts.

1 Freddie Deboer is guilty of this. Also Ed Zitron.

2 “But if you have no certainty, shouldn’t you nothing will happen?” No! Uncertainty does not asymmetrically favor the status quo! Reread the hurricane example!

 

More writing on my substack!

5

0
0

Reactions

0
0

More posts like this

Comments
No comments on this post yet.
Be the first to respond.
Curated and popular this week
Relevant opportunities