I'd be much happier if the new & upvoted were the default. I deeply dislike having the images on the front-page. 

It totally ruins the experience for me.

Thanks for posting this. It nudged us (at ML4Good) to publish our impact assessment yesterday - we previously used this for a funding application, but didn't publish openly. 

We've covered some of the criteria that you mention

• We have:
  • Published a response rate for the survey behind our outcome claims and tried to extrapolate to non responders
  • Published a fully loaded cost per outcome 
  • Pre-specified our outcome metrics before our alumni survey
  • Published methodological caveats alongside our headline figure 
• We haven't:
  • Published any comparison group, control or rejected-applicant follow-up - we will aim to do that for our next cohort.
  • Pre-specified our outcome metrics before our cohorts run

What would it mean for instrumental convergence to be "false" in this context? 

I don't think I can imagine such a world.

Good question, but I implore you not to take this post too seriously. It's a real phenomenon, but it's a real stretch to claim that this applies in the implied way to cause areas like AI safety. 

The model in the article is just a toy model of a world where existential threats are randomly distributed according to a power law, where genuinely high-probability threats are, by assumption, basically absent from the space of possible threats, and where there's no process of updating based on evidence. 

A more narrow claim like: "single narrowly defined x-risk estimates of genuinely speculative, unresearched causes are likely to be inflated" might be valid, but that doesn't seem to be what titotal is implying. He's making a claim way beyond anything implied by the model, even if the model were a valid representation of the phenomenon. He seems to believe that almost all of today's concern for AI risk is all downstream of a belief cultivated within a narrow subcommunity subject to the optimiser's curse - an extraordinary claim that requires a lot more evidence than that supplied in the article. 

The claim being made is something like:

• Some time in the 2000s, Eliezer Yudkowsky and friends made up some numbers for AI risk
• Community dynamics (rather than the merits of the arguments) spread these numbers from Bostrom to Tegmark to Sam Harris to Elon Musk to the EA community etc.
• These social dynamics had such an effect that multiple seemingly independent and unrelated people/experts from Geoffrey Hinton to Yoshua Bengio to Chinese academics and tech people from completely different intellectual lineages have absorbed this false belief from the cultural milieu (again, not at all based on the merits of the arguments), leading many of these people, as well as specialists across unrelated fields, to make estimates of AI x-risk that remain orders of magnitude too high

Note that this requires some pretty wild, difficult-to-justify assumptions on how this belief has spread.

The opposing narrative (which I would advocate for) is that:

• AI x-risk is something that has independently been identified as a risk by multiple people - often far before quantifying or ranking risks.
• The spread of these beliefs was obviously affected by community dynamics, but people largely adopted somewhat independent beliefs based on the merits of rational arguments
• Quantitative predictions were inherently imprecise because they're so dependent on messy world models, but they were grounded enough in reason and evidence, and composed of enough independent estimates, that any optimiser's curse is massively weakened
• As certain predictions came to pass, and current LLMs approach AGI in non-ideal geopolitical and competitive circumstances, this is increasingly being seen by a wider range of thinkers as a >1% x-risk
• Even if there were "optimiser curse" risks in initial prioritisation of AI, it's now increasingly recognised that AI will be a massive deal. AI-assisted engineered pandemic uplift work, observed cyber-capabilities, and signs of misalignment/scheming etc. are building on the strong theoretical evidence base that AI-generated catastrophe is possible. 

And on your particular question of how to act, even given the optimiser's curse as stated in the toy model, working on the speculative thing could still be optimal. If a highly uncertain intervention seems exceptionally promising or x-risky, the value of information becomes incredibly high, because accurate or well-reasoned research will lead to this intervention being prioritised or not by far more people. If you have a research focus, it's therefore probably more recommended to focus on a more uncertain, "high-risk-high-reward" area. You could also draw up a toy model for explore/exploit based on the optimiser's curse.

Finally, you don't necessarily have to "pick" from a narrow set of pre-defined cause areas. You can also divide or merge risks, cause areas, skill-sets etc. to be more robust, precise, or coherent with your own world model (e.g. focusing on engineered pandemics because you realise this could interact with AI-related x-risk, GCBRs, and global health).

Your piece went kinda viral on Substack, and most of the comments on the comments thread, and elsewhere on Substack, were constructive and positive. This seems a moderate win for alternative/social media.

I would barely update at all on this paper. Effect sizes are very small and don't show a logical causal mechanism.

Looking at the full paper, in the fully adjusted matched model, smaller tattoos (under one hand palm) give you a higher risk (1.27) than larger ones (1.14). In the unmatched model, large tattoos had no effect. This seems contradictory with their suggested mechanism (ink affecting lymph nodes). 

And their results imply that it's most dangerous getting tattoos before there's a plausible chemical pathway to harm (risk is highest for those who got tattoos under 2 years ago). They say this could be a result of antagonising existing tumours, but seems likely to be a statistical artifact or a non causal correlation.

If you have other evidence, though, I'd be happy to hear it.

I wrote a (draft amnesty) post on a very small subset of this - what global health charities you should donate to given different worldviews.

I agree with the "you don't have to debate on their terms" point here, but I think for 99% of your readers/listeners, it cuts far more strongly in a different way than that you're implying. 

The debate has generally been set in terms of "Anthropic vs. DoW", and, while I know zero people in our community who have taken the government's side on this, I've seen many EAs and adjacent people become increasingly uncritical supporters of Anthropic, just because they're standing against the obviously bad actor in this situation. 

I think it's important to remember:

1. If you thought Anthropic was at least somewhat untrustworthy before this, you shouldn't update too much the other way - especially when they backtracked on their RSP over the same period.
2. If you thought that Anthropic's decision to join the race towards AGI was perilous, you shouldn't really update your view on this based on the Pentagon being absurd and unpredictable.
3. Regardless of the intention and character of the government actors, it's potentially still a worrying sign that the most powerful state in the world has tried to shut down a frontier AI lab and failed spectacularly.
4. The growth and popularity of Anthropic and Claude Code have since caused the AI 2027 team to shorten their AGI timelines.