Let’s imagine it’s November 2031. We’re both still here, AI is much more capable than it is today, but governments are governing, civilisation is still (mostly) civilised, and there has been no catastrophic loss of control. It just kind of didn’t happen. 

What happens to the AI safety field then? 

I expect people who currently believe catastrophe could happen within two or three years will start saying five or ten years. Then 15. Then ‘who cares if we’re a few decades off; we’re talking about human extinction!’ Meanwhile, people who have spent years warning that we could all die soon might remind everyone that they only said could.

For people who are very scared, AI x-risk seems to have become an organising principle of their lives. Careers, financial decisions (not to mention funding decisions), whether to have children, whether to move to SF - influenced by the belief that humanity might have only a few years left.

If that ends up not being the case, the overriding temptation might be to say ‘it’s still just around the corner and, given the stakes, it’s dumb of you to expect a clear cut off point’, keep sounding the alarm for as long as funding lasts, and then… transition into something else.

…

I’d like to see more people putting numbers and dates against their predictions. Then let’s revisit those predictions in five years. EA prides itself on intellectual honesty and updating beliefs in response to evidence etc. Will those standards be applied as much to AI safety organisations as they are to most everything else?

At this point it’s probably obvious that I’m having a hard time understanding the threats are as dire as some people say. But I can’t make a serious plaidoyer for or against a belief in catastrophic AI risk, because I’m uninformed. ‘Huh, tell us something we don’t know, given this post’, some might say. Fair enough. I’ve done a couple of BlueDot courses, read around the subject, and I’m cognisant of how much the experts disagree, which makes it hard to feel conviction in either direction. Yet I hope you’ll still consider it a worthy question, no matter who’s asking. A necessary question.

What would it take for the people who are both informed and scared, to change their minds? 

46

7
1
1
1
1

Reactions

7
1
1
1
1
Comments22
Sorted by Click to highlight new comments since:

I think this question is pretty hard for me to answer because it's hard for me to imagine what the future would look like if we have AIs that are actually much more capable than today but the world mostly stays normal. I think it's much more plausible to me that I'm wrong because AIs don't become much capable, than because they do become much more capable and nothing external much changes.

  1. If I'm imagining that world, the most likely hypothesis that comes to mind is that we had some significant global governance to either slow down further AI development or to manage the risks imposed by the existing very powerful models (likely both).
    1. In such a world, whether I should work on AI safety or not depends a lot on the details of whether the global governance is in a good place and/or benefits from my help.
  2. Another possibility (that seems less likely to me but not impossible) is that we were wrong about the difficulty of aligning AI and we have some form of "alignment by default" world where the AIs are actually just much more cognitively capable than us but they're aligned to our intentions.
    1. In that world, what I should work on is in some sense not up to me (and certainly not up to my 2026-self!). I should instead be asking the new and better AIs what I should be doing, to the extent my efforts matter at all.
  3. A third possibility is that capabilities stalled at a level slightly higher than today's, but not generally dangerous (for example if AI conquered a few fields the way it conquered math and cybersecurity, but otherwise stopped). And we've seen a few years with minimal further progress. In particular, this is a world where we have convincing evidence that AI R&D is not automatable, robotics has stalled out, human persuasion and long-run autonomy seems to have stalled, etc.
    1. In such a world I think probably some people should still work on AI safety but mostly it's manageable. Maybe people should go back to theory and conceptual stuff, if it looks clear that prosaic AI won't get to true AGI/ASI. Probably I'd personally work on something else. I'm not sure.
  4. A fourth possibility is that AI capabilities are at an insanely high level and still growing, and alignment and governance still are far from solved, but we aren't yet doomed. In that world (which I think is the one you're most worried about/think is most central?) I basically think it's reasonable to think AI x-risk is just around the corner and we should still be working really hard on it.
    1. If we had, say, 3-5 more years of capabilities with AIs that are superhumanly good at science, long-term planning, real-world robotics (or side channels like persuasion and nanotech), but no major catastrophes, than I'm probably wrong about this for some key reason. #1 or #2 are the most obvious that comes to mind but maybe there's a secret third thing here that I didn't account for.

What about AI becoming unsafe at a normal you-can-be-sued-for-this level strongly enough that you don't really get much forward progress because nobody can release or use their models, and this drives some combo of an AI financial collapse and a shift towards specialized models/harnesses/other anti-bitter lesson stuff? Does this scenario put too much trust in the AI companies?

Right, so to be a bit more concrete, maybe you're imagining that after HuggingFace 3.0 the push for generalized AIs stall out and we see some impressive specialized innovations like (say) self-driving cars and specialized biology models but no more general advances? Seems plausible.

Such a world I think look a lot like some combination of my #1 and #3, except with probably less available funding for AI safety than those worlds (though likely still higher than 2025 and 2026).

Thanks for mapping those scenarios out and yes, I'm most curious about 4. I suppose I’m interested in whether there’s a conceivable future in which the field accepts that it overestimated the danger - without immediately retreating to its predictions just being premature. 

Good post. It's worth mentioning that this already happened. This is what the Singularity Institute, now called the Machine Intelligence Research Institute (MIRI), wrote in 2000. MIRI predicted that transhuman AI — what we'd now call artificial general intelligence — would be invented no later than 2020, and probably much earlier:

Our best guess for the timescale is that our final-stage AI will reach transhumanity sometime between 2005 and 2020, probably around 2008 or 2010. As always with basic research, this is only a guess, and heavily contingent on funding levels.

In 1999, Eliezer Yudkowsky, MIRI's founder, wrote that he believed dangerous nanotechnology would wipe out humanity unless we (or, specifically he) invented friendly AI first. (Now he says don't just slow down, stop. Back then he said we had to go fast.) Nowadays, we'd call friendly AI something like aligned AGI or safe AGI. Yudkowsky's name for his friendly AI project was "Ellison".

He wrote:

My current estimate, as of right now, is that humanity has no more than a 30% chance of making it, probably less. The most realistic estimate for a seed AI transcendence is 2020; nanowar, before 2015. The most optimistic estimate for project Elisson would be 2006; the earliest nanowar, 2003.

So, what happens when people get the timeline wrong by at least 6 years? You've seen one example of what has happened! Look at MIRI and at Eliezer Yudkowsky, and at the LessWrong community.

Another example I personally found interesting to look at is the eschatology of Jehovah's Witnesses. In the 1960s, the Jehovah's Witnesses leadership embraced an end of the world prophecy with a specific year attached: 1975.

This prophecy was widely taught and believed among Jehovah's Witnesses. People took this deadly seriously and did things like sell their house and devote themselves full-time to evangelism.

When 1976 rolled around and the world was not over, the church leadership backpedaled and did revisionist history on what it had actually said. Recruitment suffered greatly for at least a few years, although the church did not collapse and obviously still exists. One leader even said the prophecy's falsification was a good thing, since it separated the true faithful wheat from the wavering chaff.

Would it be so surprising if the most devoted Singularity believers kept kicking the can down the road over and over forever until they eventually died of old age 50 or 60 years from now? Not to me, at least. That seems perfectly human.

On the other hand, many people whose belief is not as strong will probably rethink things when the AI bubble pops.

Thanks Yarrow, this is fascinating historical context. I wasn’t aware of those early MIRI predictions.

I first learned about these predictions from a post on the philosopher David Thorstad's fantastic blog Reflective Altruism. Thorstad, in turn, cited this post on the EA Forum. Credit where credit is due!

There is some background that will unfortunately take a while to explain, but it seems worthwhile.

Eliezer Yudkowsky has a very odd approach where he completely changes his mind on big topics, but never really explains why, if he even acknowledges that it happened. His fear that nanotech would imminently end the world is one. Unless I'm missing something, he never explained why he changed his mind on this, besides a few short oblique comments.

You'd think it would be a big deal. He used to say that we need to rush to build AGI as fast as possible to stop the world from imminent destruction from nanotech. Now he says countries should agree to stop working on AGI and blow up the datacentres of any countries that don't comply. That's a huge shift! What happened? There's no real explanation that I can find...[1]

Another one is deep learning or deep neural networks. In 2008, Yudkowsky wrote a post that was critical of neural networks. Even into the mid-2010s, several years into the deep learning revolution that kicked off in 2012, MIRI was still primarily focused on hand-coded AI. I think it wasn't until after AlphaGo defeated Lee Sedol in 2016 that MIRI finally hired its first research intern to focus on machine learning. By no later than 2023, Yudkowsky was fully bought-in to deep learning. But what happened? When and why did he change his mind?

I've searched extensively and asked around. Nobody seems to have a clear answer on this. Even people from MIRI or Lightcone Infrastructure (the organization that runs LessWrong) don't seem to know. A member of the LessWrong community — presumably a friendly audience — wrote a critical review of the book Yudkowsky co-authored, focusing mainly on this point. Yudkowsky's talking points about AI safety are essentially the same as they were in 2008 when he was skeptical of neural networks. They have been cosmetically updated for the deep learning era. But he assumed in 2008 (and seemingly for some time after) that AGI would be hand-coded. Why should ideas about hand-coded AI still apply to deep learning? If they still apply, why do they? What's the reasoning? What's the argument? Yudkowsky never says.

The unfortunate situation is that LessWrong is essentially an Eliezer Yudkowsky fan club founded by Eliezer Yudkowsky. Yudkowsky's blog posts called The Sequences are a revered text. Even his Harry Potter fanfic is a revered text. Yudkowsky claims he is the smartest person in the world. Many of his fans seem to agree.

In a normal context, people would evenhandedly evaluate Yudkowsky's views and track record on topics like AI and nanotech. But he has many followers who see him as some kind of one-of-a-kind world-historical genius, some kind of nearly infallible mystical guru figure. He's treated with reverence and loyalty rather than critical thinking and accountability. Not by everyone, but by a lot of people.

I don't buy EA's self-congratulatory self-image, which you described:

EA prides itself on intellectual honesty and updating beliefs in response to evidence etc.

There are many reasons I don't buy this self-image, some of which I've detailed here and here. You can look at the Yudkowsky fan club as one example (among many) of why that self-image doesn't add up.


  1. Added Thursday, November 5th, 2026 at 11:57 PM Eastern: I did another cursory check on this, including the link that Matrice Jacobine shared below. I still can't find any explanation from Eliezer Yudkowsky about why he stopped believing that nanotech had a 70%+ chance of wiping out humanity by 2015. If anyone is aware of an explanation from Yudkowsky about why he changed his mind on this prediction, please let me know. ↩︎

But he has many followers who see him as some kind of one-of-a-kind world-historical genius, some kind of nearly infallible mystical guru figure.


I'd be surprised if it's as many people as you think. Most rationalists I've talked to think Eliezer has a bit too much ego and that he is a bit too confident about many of his predictions.

For what it's worth, I think regardless of whether he ends up being right or wrong, at this stage he's clearly a world-historical figure. I think he is a genius, but I also think that someone being a genius isn't as valauble as it's often portrayed. Intelligence is not the same as wisdom and, as a result, someone's extraordinary intelligence can end up being a liability, rather than an asset.

Thanks for backing up my point, I guess...

Loved the reflection! 

I would say caring for s-risks and x-risks will be still important and worth the funding and effort (the magnitude deserves it, even with a slight likelihood). 

However we should also care for more immediate and factual risks we usually leave to AI Ethics (e.g. capitalism surveillance, algorithmic bias, suicidal prevention), that could become a core part of AI Safety as well in the scenario you present.

What is (almost) certain by late 2026 is that AI is going to play a massive transformative role into society. Billions of human beings are already interacting with LLMs, AI agents are being deployed at a rapid scale, robotics is a few years ahead... 

AI Safety may need to adapt depending on how the risks scenarios unfold, but our industry is going to play an enormous role in doing the most good in the upcoming future, no matter when and if s-risks and x-risks happen. 

At least this is my view, and what motivates me to contribute to the field. I will be looking forward to hearing any thoughts you have about this! =)

What is (almost) certain by late 2026 is that AI is going to play a massive transformative role into society. Billions of human beings are already interacting with LLMs, AI agents are being deployed at a rapid scale, robotics is a few years ahead...

Wanna bet? :)

I have an open offer to bet anyone who'd like that AI is in a bubble and that the bubble will pop within the next ~7 years.

The timeframe is so long because bubbles are impossible to time precisely. Hopefully the true outcome will not take so long to become clear. But you never know.

The purpose of the bet is to do just what Siobhan said in this post:

I’d like to see more people putting numbers and dates against their predictions. Then let’s revisit those predictions in five years.

Plus, betting is fun!

Hey! Saw your post a few days ago.

My guess without betting (mostly because I don´t like to bet): There is indeed a financial bubble around AI, but AI is going to play a massive role in the transformation of society in parallel. Both statements can be true.

How? Reflect on what happen in the dot-com bubble around 1999-2000. Billions of dollars and financial assets were lost in the "internet bubble", but in the following years that same internet has completely transformed our lives.

My guess is that AI is becoming so strategically important for national security and power exertion that governments are going to step and finance it (I see US mimicking a "China approach" to AI development, at least in this field). 

That is my "bet no-bet". Really is similar than you about the bubble, but different in the development and impact of AI. =) 

With the dot-com bubble, there was a lag of about 10 years between when the bubble popped and companies like Amazon recovered to their bubble-era stock prices. How long do you think the lag with large language models will be? 5 years, 10 years, 20 years, 30 years...?

Personally, I just don't see LLMs ever becoming useful enough to justify $5 trillion in market cap or whatever it is. They aren't that useful now and they aren't improving that much. As a technology, they seem like a dead end.

The term AI is incredibly broad. It was coined in 1955 and has been used to describe many different kinds of science and technology since then. If science and technology continue to make progress, then it seems likely that eventually some technology that can rightly be called AI will have a huge, durable economic impact — whether that's in 20 years or 120 years or 200 years. But LLMs most likely won't ever have that kind of impact, even 20 years from now.

Thanks Sergio! If the probability of catastrophe falls enough, I’d hope to see a change in how resources are allocated, rather than broadening the definition of AI safety to encompass everything else.

This is a great question! 

I only decided to work on AI safety in the last year or so, largely because the current rate of progress convinced me there was a high enough chance that the next 1-5 years are pivotal. Thinking about what I'll be working on in 2031 assumes (1) we're alive and (2) humans have retained enough control over the world for my actions to matter. The answer entirely depends on why (1) and (2) are true. 

I can think of a few scenarios: 

  1. We're alive and in control in 2031 because we rose to the challenge and created some sort of (likely international) regulatory regime and/or solved the alignment problem. What I do depends on how much maintenance the solution requires, i.e. if we appear to be in a stable equilibrium. My best guess is that we won't be and there will still be lots of work to be done to maintain the fragile peace, so to speak.
  2. The tech fizzles; models plateau around the current frontier or slightly past it. AI would still be a huge deal in this world (the frontier is currently internal models that can solve major unsolved math problems), but catastrophe seems further off. This is the world where I'm most likely to pivot to working on something else. I'd keep an eye on things and possibly go back to working in AI safety if a new paradigm takes off and it looks like we're in crunch time again.
  3. Funding for scaling temporarily dries up. The Anthropic IPO is a bust, Wall St gets very nervous about the AI buildout, and some hyperscalers go under. The remaining frontier labs have a harder time convincing investors that $1B+ training runs are worth it and the rate of AI progress slows dramatically. I'm probably still working on AI safety in this world because nothing has happened to show that the underlying technology doesn't pose an existential risk. This future still has a button that maybe ends the world that anyone can pay $100B to press. The fact that no one is currently willing to pay $100B (or whatever the actual number is) to press the button doesn't change the fact that the world ending button is still waiting around for someone to press it. 

These predictions are all admittedly qualitative. I did have a quantitative prediction that played a big role in convincing me to pivot to AI safety - in mid-2024, I said that I would pivot if the METR long-horizon tasks trend line held through 2026. I updated a lot when that benchmark was saturated in the middle of this year. At some point I'll need to think about a comparable metric I can use to signal that it's time to pivot off AI safety. I don't feel a lot of urgency to do that though, given the way things are going. 

The best I have right now is "2027 and 2028 feel significantly less crazy than 2026." That's still vibes based, but feels tangible enough. I wouldn't automatically stop working on AI safety if this happened, but it would prompt me to think a lot harder about if I should be.

Thanks, I like your suggestion that 2027 and 2028 feeling less crazy than 2026 might be a useful signal, even if it’s not particularly scientific! I weakly predict 3 to be more or less what happens in terms of funding. 

Let’s imagine it’s November 2031. We’re both still here, AI is much more capable than it is today, but governments are governing, civilisation is still (mostly) civilised, and there has been no catastrophic loss of control. It just kind of didn’t happen.

Depends on what "much more capable" means. But on the sharp left turn worldview, or a worldview similar to AI 2027, this is what the world is expected to look like shortly before AI kills everyone. When misaligned AI is smart enough to be transformative but not yet powerful enough to reliably seize control, its best strategy is to pretend to be aligned until it has enough power to wipe out humanity with minimal opposition.

The point at which I'd be less worried is where AI is clearly powerful enough to kill everyone if it wants to, and it's had plenty of time to do that, but it hasn't.

How would we distinguish between those possibilities? If AI is biding it's time, we could unknowingly be in that state indefinitely... Thanks for giving a clear point at which you'd be less worried. 

How do you tell the difference between an honest person, and an extremely good liar who is also much smarter than you? I don't know if you can.

Yes, that would probably be sufficient for me to change my mind. I think I a central example of someone who is deep into AI Safety.

I assign 5% probability to the scenario you are outlining - nothing dramatic happening until 2031.

Conditional on that scenario actually happening, I am 80% certain that I would dramatically reduce my involvement in AI Safety.

Curated and popular this week
Relevant opportunities