I'm not convinced the recent capabilities advancements (highly concentrated in pure math, the section which by definition can't be applied) support the forecasts that misaligned superintelligence is on balance harmful. In general, I think there is not enough evidence to draw the conclusion that multi-agent alignment (which was far from inevitable ex-ante in comparison to the sometimes monotheist conceptions of ASI) is impossible, especially with AI eventually automating and enforcing the institutional mechanisms.
However, this is not guaranteed either. The labs are making the biggest bet that humanity has taken by far; that RSI will automate and solve the alignment problem, and the ensuing intelligence explosion will solve most of humanity's suffering. This explains much of the race dynamics until now, where the balance of tail-risks are now undoubtedly on the upside. Note also the competition with China, which plays particular relevance for open-source.
My p(doom) probably sits somewhere just over halfway (to account for the right-tailed distribution here) between my pre and (initial) post Hugging-Face likelihoods: perhaps 8%. It would be unwise not to update since this saga unfolded.
Some of the latest macroeconomic models from the likes of Anthropic and Acemoglu suggest rather grim futures for employment relative to what economists were previously predicting, hence its fair to say our methods lean overly conservative towards excess rigour here. Perhaps the recent backlash from the mathematicians, and some of the sentiment behind the anti data-centre movement, reflect these (increasingly justified) anxieties.
Nonetheless, this is orders of magnitude more dangerous than nuclear weapons. That's enough for me to take AI-safety incredibly seriously!