Let’s imagine it’s November 2031. We’re both still here, AI is much more capable than it is today, but governments are governing, civilisation is still (mostly) civilised, and there has been no catastrophic loss of control. It just kind of didn’t happen.
What happens to the AI safety field then?
I expect people who currently believe catastrophe could happen within two or three years will start saying five or ten years. Then 15. Then ‘who cares if we’re a few decades off; we’re talking about human extinction!’ Meanwhile, people who have spent years warning that we could all die soon might remind everyone that they only said could.
For people who are very scared, AI x-risk seems to have become an organising principle of their lives. Careers, financial decisions (not to mention funding decisions), whether to have children, whether to move to SF - influenced by the belief that humanity might have only a few years left.
If that ends up not being the case, the overriding temptation might be to say ‘it’s still just around the corner and, given the stakes, it’s dumb of you to expect a clear cut off point’, keep sounding the alarm for as long as funding lasts, and then… transition into something else.
…
I’d like to see more people putting numbers and dates against their predictions. Then let’s revisit those predictions in five years. EA prides itself on intellectual honesty and updating beliefs in response to evidence etc. Will those standards be applied as much to AI safety organisations as they are to most everything else?
At this point it’s probably obvious that I’m having a hard time understanding the threats are as dire as some people say. But I can’t make a serious plaidoyer for or against a belief in catastrophic AI risk, because I’m uninformed. ‘Huh, tell us something we don’t know, given this post’, some might say. Fair enough. I’ve done a couple of BlueDot courses, read around the subject, and I’m cognisant of how much the experts disagree, which makes it hard to feel conviction in either direction. Yet I hope you’ll still consider it a worthy question, no matter who’s asking. A necessary question.
What would it take for the people who are both informed and scared, to change their minds?
I think this question is pretty hard for me to answer because it's hard for me to imagine what the future would look like if we have AIs that are actually much more capable than today but the world mostly stays normal. I think it's much more plausible to me that I'm wrong because AIs don't become much capable, than because they do become much more capable and nothing external much changes.
What about AI becoming unsafe at a normal you-can-be-sued-for-this level strongly enough that you don't really get much forward progress because nobody can release or use their models, and this drives some combo of an AI financial collapse and a shift towards specialized models/harnesses/other anti-bitter lesson stuff? Does this scenario put too much trust in the AI companies?
Right, so to be a bit more concrete, maybe you're imagining that after HuggingFace 3.0 the push for generalized AIs stall out and we see some impressive specialized innovations like (say) self-driving cars and specialized biology models but no more general advances? Seems plausible.
Such a world I think look a lot like some combination of my #1 and #3, except with probably less available funding for AI safety than those worlds (though likely still higher than 2025 and 2026).
Thanks for mapping those scenarios out and yes, I'm most curious about 4. I suppose I’m interested in whether there’s a conceivable future in which the field accepts that it overestimated the danger - without immediately retreating to its predictions just being premature.
Good post. It's worth mentioning that this already happened. This is what the Singularity Institute, now called the Machine Intelligence Research Institute (MIRI), wrote in 2000. MIRI predicted that transhuman AI — what we'd now call artificial general intelligence — would be invented no later than 2020, and probably much earlier:
In 1999, Eliezer Yudkowsky, MIRI's founder, wrote that he believed dangerous nanotechnology would wipe out humanity unless we (or, specifically he) invented friendly AI first. (Now he says don't just slow down, stop. Back then he said we had to go fast.) Nowadays, we'd call friendly AI something like aligned AGI or safe AGI. Yudkowsky's name for his friendly AI project was "Ellison".
He wrote:
So, what happens when people get the timeline wrong by at least 6 years? You've seen one example of what has happened! Look at MIRI and at Eliezer Yudkowsky, and at the LessWrong community.
Another example I personally found interesting to look at is the eschatology of Jehovah's Witnesses. In the 1960s, the Jehovah's Witnesses leadership embraced an end of the world prophecy with a specific year attached: 1975.
This prophecy was widely taught and believed among Jehovah's Witnesses. People took this deadly seriously and did things like sell their house and devote themselves full-time to evangelism.
When 1976 rolled around and the world was not over, the church leadership backpedaled and did revisionist history on what it had actually said. Recruitment suffered greatly for at least a few years, although the church did not collapse and obviously still exists. One leader even said the prophecy's falsification was a good thing, since it separated the true faithful wheat from the wavering chaff.
Would it be so surprising if the most devoted Singularity believers kept kicking the can down the road over and over forever until they eventually died of old age 50 or 60 years from now? Not to me, at least. That seems perfectly human.
On the other hand, many people whose belief is not as strong will probably rethink things when the AI bubble pops.
Thanks Yarrow, this is fascinating historical context. I wasn’t aware of those early MIRI predictions.
I first learned about these predictions from a post on the philosopher David Thorstad's fantastic blog Reflective Altruism. Thorstad, in turn, cited this post on the EA Forum. Credit where credit is due!
There is some background that will unfortunately take a while to explain, but it seems worthwhile.
Eliezer Yudkowsky has a very odd approach where he completely changes his mind on big topics, but never really explains why, if he even acknowledges that it happened. His fear that nanotech would imminently end the world is one. Unless I'm missing something, he never explained why he changed his mind on this, besides a few short oblique comments.
You'd think it would be a big deal. He used to say that we need to rush to build AGI as fast as possible to stop the world from imminent destruction from nanotech. Now he says countries should agree to stop working on AGI and blow up the datacentres of any countries that don't comply. That's a huge shift! What happened? There's no real explanation that I can find...[1]
Another one is deep learning or deep neural networks. In 2008, Yudkowsky wrote a post that was critical of neural networks. Even into the mid-2010s, several years into the deep learning revolution that kicked off in 2012, MIRI was still primarily focused on hand-coded AI. I think it wasn't until after AlphaGo defeated Lee Sedol in 2016 that MIRI finally hired its first research intern to focus on machine learning. By no later than 2023, Yudkowsky was fully bought-in to deep learning. But what happened? When and why did he change his mind?
I've searched extensively and asked around. Nobody seems to have a clear answer on this. Even people from MIRI or Lightcone Infrastructure (the organization that runs LessWrong) don't seem to know. A member of the LessWrong community — presumably a friendly audience — wrote a critical review of the book Yudkowsky co-authored, focusing mainly on this point. Yudkowsky's talking points about AI safety are essentially the same as they were in 2008 when he was skeptical of neural networks. They have been cosmetically updated for the deep learning era. But he assumed in 2008 (and seemingly for some time after) that AGI would be hand-coded. Why should ideas about hand-coded AI still apply to deep learning? If they still apply, why do they? What's the reasoning? What's the argument? Yudkowsky never says.
The unfortunate situation is that LessWrong is essentially an Eliezer Yudkowsky fan club founded by Eliezer Yudkowsky. Yudkowsky's blog posts called The Sequences are a revered text. Even his Harry Potter fanfic is a revered text. Yudkowsky claims he is the smartest person in the world. Many of his fans seem to agree.
In a normal context, people would evenhandedly evaluate Yudkowsky's views and track record on topics like AI and nanotech. But he has many followers who see him as some kind of one-of-a-kind world-historical genius, some kind of nearly infallible mystical guru figure. He's treated with reverence and loyalty rather than critical thinking and accountability. Not by everyone, but by a lot of people.
I don't buy EA's self-congratulatory self-image, which you described:
There are many reasons I don't buy this self-image, some of which I've detailed here and here. You can look at the Yudkowsky fan club as one example (among many) of why that self-image doesn't add up.
Added Thursday, November 5th, 2026 at 11:57 PM Eastern: I did another cursory check on this, including the link that Matrice Jacobine shared below. I still can't find any explanation from Eliezer Yudkowsky about why he stopped believing that nanotech had a 70%+ chance of wiping out humanity by 2015. If anyone is aware of an explanation from Yudkowsky about why he changed his mind on this prediction, please let me know. ↩︎
I'd be surprised if it's as many people as you think. Most rationalists I've talked to think Eliezer has a bit too much ego and that he is a bit too confident about many of his predictions.
For what it's worth, I think regardless of whether he ends up being right or wrong, at this stage he's clearly a world-historical figure. I think he is a genius, but I also think that someone being a genius isn't as valauble as it's often portrayed. Intelligence is not the same as wisdom and, as a result, someone's extraordinary intelligence can end up being a liability, rather than an asset.
Thanks for backing up my point, I guess...
Loved the reflection!
I would say caring for s-risks and x-risks will be still important and worth the funding and effort (the magnitude deserves it, even with a slight likelihood).
However we should also care for more immediate and factual risks we usually leave to AI Ethics (e.g. capitalism surveillance, algorithmic bias, suicidal prevention), that could become a core part of AI Safety as well in the scenario you present.
What is (almost) certain by late 2026 is that AI is going to play a massive transformative role into society. Billions of human beings are already interacting with LLMs, AI agents are being deployed at a rapid scale, robotics is a few years ahead...
AI Safety may need to adapt depending on how the risks scenarios unfold, but our industry is going to play an enormous role in doing the most good in the upcoming future, no matter when and if s-risks and x-risks happen.
At least this is my view, and what motivates me to contribute to the field. I will be looking forward to hearing any thoughts you have about this! =)
Wanna bet? :)
I have an open offer to bet anyone who'd like that AI is in a bubble and that the bubble will pop within the next ~7 years.
The timeframe is so long because bubbles are impossible to time precisely. Hopefully the true outcome will not take so long to become clear. But you never know.
The purpose of the bet is to do just what Siobhan said in this post:
Plus, betting is fun!
Hey! Saw your post a few days ago.
My guess without betting (mostly because I don´t like to bet): There is indeed a financial bubble around AI, but AI is going to play a massive role in the transformation of society in parallel. Both statements can be true.
How? Reflect on what happen in the dot-com bubble around 1999-2000. Billions of dollars and financial assets were lost in the "internet bubble", but in the following years that same internet has completely transformed our lives.
My guess is that AI is becoming so strategically important for national security and power exertion that governments are going to step and finance it (I see US mimicking a "China approach" to AI development, at least in this field).
That is my "bet no-bet". Really is similar than you about the bubble, but different in the development and impact of AI. =)
With the dot-com bubble, there was a lag of about 10 years between when the bubble popped and companies like Amazon recovered to their bubble-era stock prices. How long do you think the lag with large language models will be? 5 years, 10 years, 20 years, 30 years...?
Personally, I just don't see LLMs ever becoming useful enough to justify $5 trillion in market cap or whatever it is. They aren't that useful now and they aren't improving that much. As a technology, they seem like a dead end.
The term AI is incredibly broad. It was coined in 1955 and has been used to describe many different kinds of science and technology since then. If science and technology continue to make progress, then it seems likely that eventually some technology that can rightly be called AI will have a huge, durable economic impact — whether that's in 20 years or 120 years or 200 years. But LLMs most likely won't ever have that kind of impact, even 20 years from now.
Thanks Sergio! If the probability of catastrophe falls enough, I’d hope to see a change in how resources are allocated, rather than broadening the definition of AI safety to encompass everything else.
This is a great question!
I only decided to work on AI safety in the last year or so, largely because the current rate of progress convinced me there was a high enough chance that the next 1-5 years are pivotal. Thinking about what I'll be working on in 2031 assumes (1) we're alive and (2) humans have retained enough control over the world for my actions to matter. The answer entirely depends on why (1) and (2) are true.
I can think of a few scenarios:
These predictions are all admittedly qualitative. I did have a quantitative prediction that played a big role in convincing me to pivot to AI safety - in mid-2024, I said that I would pivot if the METR long-horizon tasks trend line held through 2026. I updated a lot when that benchmark was saturated in the middle of this year. At some point I'll need to think about a comparable metric I can use to signal that it's time to pivot off AI safety. I don't feel a lot of urgency to do that though, given the way things are going.
The best I have right now is "2027 and 2028 feel significantly less crazy than 2026." That's still vibes based, but feels tangible enough. I wouldn't automatically stop working on AI safety if this happened, but it would prompt me to think a lot harder about if I should be.
Thanks, I like your suggestion that 2027 and 2028 feeling less crazy than 2026 might be a useful signal, even if it’s not particularly scientific! I weakly predict 3 to be more or less what happens in terms of funding.
Depends on what "much more capable" means. But on the sharp left turn worldview, or a worldview similar to AI 2027, this is what the world is expected to look like shortly before AI kills everyone. When misaligned AI is smart enough to be transformative but not yet powerful enough to reliably seize control, its best strategy is to pretend to be aligned until it has enough power to wipe out humanity with minimal opposition.
The point at which I'd be less worried is where AI is clearly powerful enough to kill everyone if it wants to, and it's had plenty of time to do that, but it hasn't.
How would we distinguish between those possibilities? If AI is biding it's time, we could unknowingly be in that state indefinitely... Thanks for giving a clear point at which you'd be less worried.
How do you tell the difference between an honest person, and an extremely good liar who is also much smarter than you? I don't know if you can.
Yes, that would probably be sufficient for me to change my mind. I think I a central example of someone who is deep into AI Safety.
I assign 5% probability to the scenario you are outlining - nothing dramatic happening until 2031.
Conditional on that scenario actually happening, I am 80% certain that I would dramatically reduce my involvement in AI Safety.