I think people continue to underrate the value of writing the clearest, most basic, simplest possible case for important parts of AI safety. For example, before I wrote my own explainer for the simplest possible case for AI doom, I noticed that all existing explainers, of which there aren't many, had multiple of the six problems below:
- They have too many extraneous details
- They appeal to overly complex analogies
- They seem to spend much of their time responding to insider debates and come across as shadow-boxing objections.
- They have "meta" stuff prominently in the text.
- Often they "sound like science fiction." I think this plausibly was correct historically but in the year 2026 they don't need to be.
- Often they reference too much insider jargon and language.
I set out to improve on the SOTA. I did not fully succeed[1], but I think it was moderately better. Since then, there have been other, probably better, explainers (Yay!).
But actually this is true not just for the core case but many other parts of relevant safety arguments as well. For example, before I wrote my explainer on recurrent neuralese (which has since been superseded by Scott's better explainer), the top Google search results for "neuralese" was some combination of a hacker news thread on AI 2027, AI-generated blogs and some tweet that confuses more than it enlightens. Oops.
But maybe neuralese isn't just that big a deal? Well we also don't have a good longform post explaining the details/mechanics of how AIs can actually kill everyone (this may sound obvious to people here, but it's patently not obvious to the general public[2]!)
We also don't have great explanations for what RSI even is, never mind why and how to ban it. In my own research area, AI superpersuasion, I see many areas of confusion regularly[3]. And so forth.
A lot of the time I think people just don't try to explain things clearly and often. In 2019, back when I lived in Silicon Valley, I'd talk to my programmer friends and often see them thinking they understand the case for risk but totally not getting basic concepts. So I'd try to explain what I believe in simple technical concepts that are graspable to programmers. I did it a few times and people quite liked it/were moderately convinced. I asked a seniorish person in AI safety whether it was worth writing it up and they were unconvinced it'd be worth my time. He was skeptical my article would lead to much since it wouldn't move people beyond Kelsey's explainer. I think this is miscalibrated[4]:
- Different things convince different people. Kelsey's a better writer than me, and writing to a general audience. I was targeting programmers/technical talent.
- If something's sufficiently important, you want more shots on target.
- Mere exposure effect (above some quality bar) is important for something like AI doom. In 2019 (and even today), the amount the public is talking about AI risk is far below the optimal level.
- My time in 2019 wasn't that valuable anyway.
I ended up spending my free time learning more about ML, and writing up why I think climate change is overrated[5]. I enjoyed and learned from those experiences, but overall I think this was a mistake in prioritization. I believe our community is continuing to make this mistake, both in terms of the general case for AI doom and in the particulars of things like what misalignment is, various threat models, what policymakers could do, and so forth.
I briefly helped work on agi.fyi, an ai safety resources recommendations site, and definitely think it's a net positive project, but imo the biggest bottleneck in comms is the lack of good explainers, rather than the explainers failing to surface to the top.
I understand that many people have very large opportunity costs. Even so, I suspect a) this is worth it to many people who are good communicators and yet believe that their opportunity costs are quite large (for example I think Nate Soares is making the right call in spending a lot of his time on AI doom comms) and b) people with lower opportunity costs should attempt to do this despite on average expecting to produce lower quality content, and bite off different parts of the information pipeline. For example, at AI safety fellowships, people at the 90th percentile of writing/communications ability or higher should aim to spend around two weeks to write their own version of the "simplest case for AI catastrophe" as they understand it, as well as address confusions about certain concepts that have not been adequately explained. Lurkers/writers on LessWrong or the Forum who are unsure about what to do and feeling sad about being unable to contribute directly to technical AI safety research or policymaking should also consider communications as a route to positive impact.
Sometimes people are worried about low-quality comms/arguments drowning out the good ones, and/or the bad arguments inoculating you against good ones. I'm skeptical, for three reasons:
- In many cases, if you delve[6] into the details, the proper response is "what good ones?" Like there isn't a better argument to supplant!
- I think reality does a decent job of sorting out good arguments on-target from the trash. Both in terms of esteem and in terms of popularity. At least in AI safety, I expect there just aren't many good arguments that get buried by trash, as evidenced by the top of Google search results, what shows up in algorithmic news feeds, what elites talk about, etc.
- To the extent that the bad arguments drowning out the good is a real problem in the future, it probably isn't that hard to course correct and fix. I think our community has pretty natural systems for doing this.
This also applies to other mediums, like shortform text (Tweets), shortform video, longform video, podcasts, speeches, and so forth[7]. It's good to persist and keep trying.
- ^
AIs flag my article as having too many inappropriate jokes and gesturing at the last step "This can all end badly" rather than going through the details. I agree, but I believe both problems are structural to my article rather than things I can easily fix with the article as-is. For example I think removing the jokes will harm both readability and theme, and wasting screen time on covering the details of how AI will kill everyone will be net less convincing (for my article specifically) than "this can all end badly."
- ^
Tbc your SOTA to beat imo is "Terminator but worse." I think this is a fine baseline, enough to get most reasonable people to be at least a bit concerned. Most explanations are in fact worse than this.
- ^
Obviously it's partially my fault for not making a "simplest case" argument somewhere for superhuman persuasion risks. I'm part of the problem!
- ^
Though the advice might accidentally have been good in that writing an explainer in 2019 targeted at programmers might've had a negative net effect on the world.
- ^
Stefan Schubert mentioned this at the time, saying something like "this is very clear and simple communication on climate change, wish more people did this for AI safety." I should've taken his points more to heart then. Oops.
- ^
What a great and underused word! I genuinely love it.
- ^
Though imo I think within AI safety comms we're slightly overinvested in being twitter reply-guys[8] and maybe podcasts vs advancing our own position clearly and forcefully. Otoh I think video and speeches are probably underrated relative to longform text. But this is micro-optimizing. The broader message of doing everything more is more critical.
- ^
My general rule of thumb in comms is that it's usually a mistake to devote time to objections, replies, etc, unless you absolutely have to. Though maybe this is because I spend too much time on replies, and the law of equal and opposite advice can apply to other people.
I agree! And even for topics where a good explainer exists, it might not be up to date / might not fit the current discourse.
Most people didn’t expect the Jacob Coxon tweet to get this level of reach, since there have been plenty of public statements about x-risk by lab insiders before. But somehow, his tweet met the moment more than anything else. I think the same can be true for explainers. Even if Kelsey Piper wrote a better version in 2019, or there’s already a 2011 Yudkowsky post explaining the same thing, a 2026 version adapted to the current situation might be what’s needed to reach and move people.
While we're requesting things, I'd like explainers to send to specific audiences. Explainers that are more than informative, but also to influence cause prioritization/political priorities relative to others
for Christians: woot @Shon Pan https://projectimagodei.org/
Re 1a, I'm happy to advise or mentor somebody interested in writing this up, I probably thought about climate change more than the vast majority of people in EA (not as much as actual experts like Johannes Ackva or John Halstead).
The ideal candidate is somebody with legible climate credentials (eg worked as a climate scientist or policymaker and is pivoting/have pivoted to AI safety.
I think I won't be a good fit for writing this up myself because I think this is one of the messages where the messenger matters and also there are cultural nuances I won't be as familiar with.
Earlier notes here (though tbc it's backed by much more than just the bullet points and I also thought about it some since).
Agreed, will be on it soon!
I agree we need more basic AI safety explainers. But I think there is a risk here too; in your own simplest case some quite big steps go very fast.
There is a correspondence problem here. We move from agentic behaviour to stable goals, then to self-preservation and power-seeking. We first need to show that these descriptions really map onto the same underlying process. Otherwise the difficult part is not explained, just skipped.For me these are not just details in the argument.
Your point about a visible failure disappearing has a documented case. In March 2025 OpenAI penalized reasoning models for “bad thoughts” in their chain of thought. Most of the misbehavior continued; what changed was that the models learned to hide their intent. The failure disappeared from the monitor, not from the model.
OpenAI now recommends keeping strong pressure off the chain of thought, so it stays readable. Your inference problem, in one experiment: what disappears from view has not yet disappeared.
Thank you, this was genuinely useful. I am building a small model around drift at the moment, and your last sentence gave me a new direction for it.
I was mainly thinking about how to distinguish different drift mechanisms, but the idea that intervention itself can change what remains observable adds another layer. I am going to extend the model to test that.
Glad it helps.
The paper behind it, for your model: Baker et al., “Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation” (OpenAI, March 2025), arXiv 2503.11926. They call the price of keeping the chain of thought readable a “monitorability tax”.
Merci beaucoup!
Hmm. To the extent that my intro isn't very good, I think it's a skill issue on my part! Some sequence of words surely must exist that conveys the same points well.
If you're interested, I'd encourage you to try to write the same article yourself here. :)
Thanks. I think I was trying to make a slightly different point. For me this is not mainly a communication problem, but an inference problem. Your wording is clear, and I did like your article on Substack. I responded to it because it did make me think about somethng I'm working on.
What could be interesting as a next step is to formalise some of the steps you describe. I tend to use maths for this. Not because the maths solves the problem, but because it forces us to say more precisely what exactly we are claiming, what the variables stand for, and where the assumptions enter.
What would count as 'agentic behaviour'? How would we measure a stable goal across different contexts? What observation would count as self-preservation, rather than just following the local incentive structure? And what would distinguish power-seeking from ordinary search, persistence, reward optimisation, or something produced by the scaffold?
If these variables are not operationalised separately, it becomes quite easy to write a chain like A → B → C and then treat the arrows as if they are already established. But for me the arrows are actually the interesting part. The fact that B can follow A in one setup does not yet tell us whether A and B are manifestations of the same underlying mechanism, or whether another mechanism can produce the same behaviour.
We would need evidence that lets us tell these different explanations apart. So for me the main question is not how to explain the argument more clearly, but what would make each step convincing. And what would count as evidence against it.
One example of the kind of thing I mean is Dawn Drescher’s recent framework on robustness and backfire risks in AI safety and governance:
https://forum.effectivealtruism.org/posts/JA68xTZmDiQFaxn73/robustness-backfire-risks-in-ai-safety-and-governance
I know this example is not exactly the point of your post, but I think her framework is useful because she separates different mechanisms instead of treating everything as one general failure mode. That still does not solve the inference problem, but it gives you more specific things to test.
A visible failure disappearing can mean that the underlying mechanism changed. But it can also mean that the same problem became less visible. At the behavioural level these may look quite similar, while causally they are very different.
At the moment I am collecting and comparing different forms of AI drift. I keep finding the same problem: quite different mechanisms can produce similar behaviour, while we use the word drift for all of them. Sometimes there really is a change in the objective. In other cases the environment or the model’s internal representation may have changed, and sometimes the only thing we have actually observed is different behaviour.
If we use the same word for all of this, it becomes difficult to know what exactly we have explained.That may be worth a post of its own. Your post did make me think more about how to make something like that accessible to people outside the field too, so thank you for that.