Thank you!

Leaving aside the criminality/corruption topic (which is significant), one potential lesson of the FTX collapse is simply around risk. As the commentator Matt Yglesias pointed out on the EA Forum years ago, there was plenty of evidence that Sam Bankman-Fried had a gambling addict's approach to risk. I'd add that there was also plenty of evidence that crypto, in general, is super risky. Yet few people in the EA community — if any — seemed to prepare for the possibility that things might go wrong, or seemed to take the level of risk seriously enough.

Another cautionary tale is the dramatic collapse of Leopold Aschenbrenner's Situational Awareness hedge fund: https://www.cnbc.com/amp/2026/07/31/leopold-aschenbrenner-situational-awareness-fund-fire-sale.html

Aschenbrenner worked at the FTX Future Fund and his super long treatise "Situational Awareness" is somewhat well-known in the EA community.

Even where there's a realistic chance to win big, it is perilous to cheerfully hand-wave away risk.

Thanks for backing up my point, I guess...

I'm still looking for someone who's willing to bet me (for a nominal amount of money for charity) on whether the AI industry is currently in a bubble that will pop within the next ~7 years: https://forum.effectivealtruism.org/posts/eCWDZb4rcCjiCtrt7/if-you-re-confident-there-s-no-ai-bubble-consider-betting-me

I think AI is in a bubble that will pop within ~7 years, so I'm looking for someone who will take the other side, who strongly believes AI is not in a bubble.

If multiple people end up accepting, that's good too, since it's a small amount of money for charity.

This is one of the more objective, more easily resolvable bets that can be made about near-term general AI capabilities progress. Moreover, unlike, say, bets on many LLM benchmarks, the outcome can't easily be gamed by AI companies in a way that's antithetical to the spirit of what the bet was trying to capture.

Check out the bet! If you don't like any of the details, propose a change and I'll seriously consider it.

Hi Matrice. I have observed a pattern in your comments where you make bold claims, often quite rudely, with citations that don't turn out to verify the claims. When pressed, you have just doubled down on things that are plainly false — about which there is no meaningful controversy among knowledgeable people, about which there is full consensus among experts, such as the definitions of certain terms in machine learning — even as I have given citations that do verify my claims.

I wish you well, but I don't find it productive to engage with this pattern. Take care. I would ask that you please refrain from trying to engage with me in the future.

If the EA Forum had blocking, I would block you. In lieu of blocking, please respect my wishes to not interact with you going forward.

I noticed something really bizarre in this post:

This is 1999 we're talking about here; almost nobody is putting probability distributions on things, that element of your later subculture has not yet been introduced. Eliezer-2002 hasn't been sent a copy of "Judgment Under Uncertainty" by Emil Gilliam. Eliezer-2006 hasn't put his draft online for "Cognitive biases potentially affecting judgment of global risks". The Sequences won't start until another year after that. How would the forerunners of effective altruism in 1999 know about putting probability distributions on forecasts? I haven't told them to do that yet! We can give historical personages credit when they seem to somehow end up doing better than their surroundings would suggest; it is unreasonable to hold them to modern standards, or expect them to have finished refining those modern standards by the age of nineteen.

Is Eliezer Yudkowsky claiming that he invented, "refined", or at least popularized the idea of "putting probability distributions on forecasts"? If so, that's just not true.

Just to pick one random example, the psychologist Philip Tetlock published a paper in 1998 based on a geopolitical forecasting exercise where participants were asked to assign a probability between 0.0 and 1.0 to a list of outcomes. The participants were told the probabilities were supposed to add up to 1.0 — these were intended as mutually exclusive outcomes.

A 1999 article in the Washington Post described some of Tetlock's research on political forecasting, so it gained some level of attention outside academia even then.

Tetlock further popularized his research on probabilistic forecasting in his 2005 book Expert Political Judgment: How Good Is it? How Can We Know?. See this review from the New Yorker that same year. Probabilistic forecasting was already in the public consciousness to some degree by 2005. Yudkowsky can't claim to have invented or "refined" it in 2006 or 2007, as he seems to imply in this strange part of this strange post. He can't even claim to have popularized it.

Yudkowsky asks:

How would the forerunners of effective altruism in 1999 know about putting probability distributions on forecasts? I haven't told them to do that yet!

In 1999, people could have known about putting probability distributions on forecasts by reading Philip Tetlock's research. They might have seen a Washington Post article about it. They certainly didn't need to wait for Eliezer Yudkowsky to introduce them to the concept.

To be clear, I don't think Tetlock invented this concept either. Tetlock's research is just one example of an idea that's been around for a long time.

This comes across as really bizarre because it's so clearly wrong. Tetlock is well-known enough that it wouldn't have been hard for Yudkowsky to know that Tetlock's work on probabilistic forecasting long predates Yudkowsky's writing on the subject. Also, in general, if you want to take credit for an idea, you should check things like that.

I first learned about these predictions from a post on the philosopher David Thorstad's fantastic blog Reflective Altruism. Thorstad, in turn, cited this post on the EA Forum. Credit where credit is due!

There is some background that will unfortunately take a while to explain, but it seems worthwhile.

Eliezer Yudkowsky has a very odd approach where he completely changes his mind on big topics, but never really explains why, if he even acknowledges that it happened. His fear that nanotech would imminently end the world is one. Unless I'm missing something, he never explained why he changed his mind on this, besides a few short oblique comments.

You'd think it would be a big deal. He used to say that we need to rush to build AGI as fast as possible to stop the world from imminent destruction from nanotech. Now he says countries should agree to stop working on AGI and blow up the datacentres of any countries that don't comply. That's a huge shift! What happened? There's no real explanation that I can find...^[1]

Another one is deep learning or deep neural networks. In 2008, Yudkowsky wrote a post that was critical of neural networks. Even into the mid-2010s, several years into the deep learning revolution that kicked off in 2012, MIRI was still primarily focused on hand-coded AI. I think it wasn't until after AlphaGo defeated Lee Sedol in 2016 that MIRI finally hired its first research intern to focus on machine learning. By no later than 2023, Yudkowsky was fully bought-in to deep learning. But what happened? When and why did he change his mind?

I've searched extensively and asked around. Nobody seems to have a clear answer on this. Even people from MIRI or Lightcone Infrastructure (the organization that runs LessWrong) don't seem to know. A member of the LessWrong community — presumably a friendly audience — wrote a critical review of the book Yudkowsky co-authored, focusing mainly on this point. Yudkowsky's talking points about AI safety are essentially the same as they were in 2008 when he was skeptical of neural networks. They have been cosmetically updated for the deep learning era. But he assumed in 2008 (and seemingly for some time after) that AGI would be hand-coded. Why should ideas about hand-coded AI still apply to deep learning? If they still apply, why do they? What's the reasoning? What's the argument? Yudkowsky never says.

The unfortunate situation is that LessWrong is essentially an Eliezer Yudkowsky fan club founded by Eliezer Yudkowsky. Yudkowsky's blog posts called The Sequences are a revered text. Even his Harry Potter fanfic is a revered text. Yudkowsky claims he is the smartest person in the world. Many of his fans seem to agree.

In a normal context, people would evenhandedly evaluate Yudkowsky's views and track record on topics like AI and nanotech. But he has many followers who see him as some kind of one-of-a-kind world-historical genius, some kind of nearly infallible mystical guru figure. He's treated with reverence and loyalty rather than critical thinking and accountability. Not by everyone, but by a lot of people.

I don't buy EA's self-congratulatory self-image, which you described:

EA prides itself on intellectual honesty and updating beliefs in response to evidence etc.

There are many reasons I don't buy this self-image, some of which I've detailed here and here. You can look at the Yudkowsky fan club as one example (among many) of why that self-image doesn't add up.

With the dot-com bubble, there was a lag of about 10 years between when the bubble popped and companies like Amazon recovered to their bubble-era stock prices. How long do you think the lag with large language models will be? 5 years, 10 years, 20 years, 30 years...?

Personally, I just don't see LLMs ever becoming useful enough to justify $5 trillion in market cap or whatever it is. They aren't that useful now and they aren't improving that much. As a technology, they seem like a dead end.

The term AI is incredibly broad. It was coined in 1955 and has been used to describe many different kinds of science and technology since then. If science and technology continue to make progress, then it seems likely that eventually some technology that can rightly be called AI will have a huge, durable economic impact — whether that's in 20 years or 120 years or 200 years. But LLMs most likely won't ever have that kind of impact, even 20 years from now.

What is (almost) certain by late 2026 is that AI is going to play a massive transformative role into society. Billions of human beings are already interacting with LLMs, AI agents are being deployed at a rapid scale, robotics is a few years ahead...

Wanna bet? :)

I have an open offer to bet anyone who'd like that AI is in a bubble and that the bubble will pop within the next ~7 years.

The timeframe is so long because bubbles are impossible to time precisely. Hopefully the true outcome will not take so long to become clear. But you never know.

The purpose of the bet is to do just what Siobhan said in this post:

I’d like to see more people putting numbers and dates against their predictions. Then let’s revisit those predictions in five years.

Plus, betting is fun!

Good post. It's worth mentioning that this already happened. This is what the Singularity Institute, now called the Machine Intelligence Research Institute (MIRI), wrote in 2000. MIRI predicted that transhuman AI — what we'd now call artificial general intelligence — would be invented no later than 2020, and probably much earlier:

Our best guess for the timescale is that our final-stage AI will reach transhumanity sometime between 2005 and 2020, probably around 2008 or 2010. As always with basic research, this is only a guess, and heavily contingent on funding levels.

In 1999, Eliezer Yudkowsky, MIRI's founder, wrote that he believed dangerous nanotechnology would wipe out humanity unless we (or, specifically he) invented friendly AI first. (Now he says don't just slow down, stop. Back then he said we had to go fast.) Nowadays, we'd call friendly AI something like aligned AGI or safe AGI. Yudkowsky's name for his friendly AI project was "Ellison".

He wrote:

My current estimate, as of right now, is that humanity has no more than a 30% chance of making it, probably less. The most realistic estimate for a seed AI transcendence is 2020; nanowar, before 2015. The most optimistic estimate for project Elisson would be 2006; the earliest nanowar, 2003.

So, what happens when people get the timeline wrong by at least 6 years? You've seen one example of what has happened! Look at MIRI and at Eliezer Yudkowsky, and at the LessWrong community.

Another example I personally found interesting to look at is the eschatology of Jehovah's Witnesses. In the 1960s, the Jehovah's Witnesses leadership embraced an end of the world prophecy with a specific year attached: 1975.

This prophecy was widely taught and believed among Jehovah's Witnesses. People took this deadly seriously and did things like sell their house and devote themselves full-time to evangelism.

When 1976 rolled around and the world was not over, the church leadership backpedaled and did revisionist history on what it had actually said. Recruitment suffered greatly for at least a few years, although the church did not collapse and obviously still exists. One leader even said the prophecy's falsification was a good thing, since it separated the true faithful wheat from the wavering chaff.

Would it be so surprising if the most devoted Singularity believers kept kicking the can down the road over and over forever until they eventually died of old age 50 or 60 years from now? Not to me, at least. That seems perfectly human.

On the other hand, many people whose belief is not as strong will probably rethink things when the AI bubble pops.