AI safety
AI safety
Studying and reducing the existential risks posed by advanced artificial intelligence

Quick takes

12
7d
2
Very low confidence in this take, but I suspect that in university groups, Better Futures fieldbuilding looks much better under short timelines than technical safety research fieldbuilding. As far as I can tell, the barriers to doing important stuff in that field are much lower. 
4
11d
2
I'm still looking for someone who's willing to bet me (for a nominal amount of money for charity) on whether the AI industry is currently in a bubble that will pop within the next ~7 years: https://forum.effectivealtruism.org/posts/eCWDZb4rcCjiCtrt7/if-you-re-confident-there-s-no-ai-bubble-consider-betting-me I think AI is in a bubble that will pop within ~7 years, so I'm looking for someone who will take the other side, who strongly believes AI is not in a bubble. If multiple people end up accepting, that's good too, since it's a small amount of money for charity. This is one of the more objective, more easily resolvable bets that can be made about near-term general AI capabilities progress. Moreover, unlike, say, bets on many LLM benchmarks, the outcome can't easily be gamed by AI companies in a way that's antithetical to the spirit of what the bet was trying to capture. Check out the bet! If you don't like any of the details, propose a change and I'll seriously consider it.
1
15d
Alignment premium and misalignment tax It’s interesting how it feels like in some ways there is or could be both of these; Alignment premium: People like Claude and some are willing to pay more for a model that feels like it is more ethical, has better character, is less sycophantic, is less likely to lie or deceive you. Misalignment tax: On the other hand, if models are badly misaligned enough, serious incidents could be increasingly expensive, both in terms of legal damages, and just as much in terms of needing to slow down or delay model releases to cope with, fix, or re-train misaligned models.
2
16d
As I have been working on SIRA (Societal Impacts Research on AI) for the past three months or so, a common sentiment has been expressed that doesn't sit well with me: that there is limited evidence of current demand for the research and talent SIRA would aim to produce. While I can see how this may be true, I'm reminded of the line attributed (probably apocryphally) to Henry Ford, "If I had asked people what they wanted, they would have said faster horses," and Steve Jobs's version, that people don't know what they want until you show it to them. I'm using this quick take as a public note to work on this tension and look for evidence against that sentiment. My plan: (1) Make a list of "societal impact" research that I can point to and say, "Hey, look, so-and-so has done this and has addressed xyz." (2) Trace the "impact" of these works to show funders/partners that this type of work can and does make an impact. (3) Write a full blog post about this tension, pointing to this evidence.
14
17d
3
Aimed at an audience newer to EA/AI safety It seems like many people have been confused lately about why 1) AI safety orgs are in desperate need of good hires and 2) org hiring rates are very low. My understanding is that there are many reasons for this, but a 2021 post states it pretty well: The best career advice seems to be to focus on developing a rare and valuable skill that orgs currently need. In AI safety, this skill is a combination of specific context, agency, mission-alignment, and strategic vision. You can develop it by reading a lot, writing out your thoughts, talking to people at events, and completing real projects. It's hard to do really impactful work. You need a plausible explanation for why your efforts are the best way your salary can be spent. This takes time, effort, and careful strategizing. Most of the world has high schools, colleges, and internships to train people for jobs, but a similar system is only recently becoming legible for AI safety and EA. The rare skills are rare for a reason. Still, I think the broader EA community could do a better job of communicating what it means for these "talent-gaps" to exist.
4
19d
1
Can anyone explain the case for why the kind of AI R&D that Anthropic is doing (i.e., building more capable frontier models) reduces X-risk? I'm aware that some people think this is the case, but I don't understand why.
2
19d
MATS Residency applications close this Friday, Oct 31. The Residency is built for experienced researchers with an established track record, ready to do independent, full-time AI safety work. Residents get: • $155k–$285k compensation • Abundant compute • Support from leading AI safety researchers If you know someone with a strong research background who's been thinking about AI safety, please send this their way. Apply: matsprogram.org/residency
11
21d
5
Anthropic just released its S-1 form, which contains the official financial information it must report to the SEC in order to go public. From the S-1, we now know that Anthropic's revenue grew more than 100x (specifically, 112x) from 2023 to 2025. Revenue was: * $41 million in 2023 * $386 million in 2024 * $4.59 billion in 2025 A few people in the EA community apparently believe this revenue growth rate will continue for the next several years at least. However, I am confident we won't see Anthropic bring in $514 billion in revenue in 2027, which is what extrapolation would imply. Indeed, I'm confident that Anthropic won't hit half of what extrapolation would imply, which is $257 billion in 2027. I'm willing to bet as much and would be delighted to wager $20 on it, to be donated to the charity of the winner's choice. I also have an open offer for a more general AI bubble bet. ---------------------------------------- Update (Friday, October 30th at 5:45pm Eastern): Here's a good article on this topic. Two economists look at Anthropic's S-1 and ask whether a $2 trillion valuation is reasonable: https://econjared.substack.com/p/the-fantastical-promises-of-a-2-trillion
Load more (8/302)