It doesn't feel that way to me but I might not be aware of all the relevant considerations, for example what jobs or other paths you're thinking of for university groups to be involved. Can you elaborate a little?

Right, so to be a bit more concrete, maybe you're imagining that after HuggingFace 3.0 the push for generalized AIs stall out and we see some impressive specialized innovations like (say) self-driving cars and specialized biology models but no more general advances? Seems plausible.

Such a world I think look a lot like some combination of my #1 and #3, except with probably less available funding for AI safety than those worlds (though likely still higher than 2025 and 2026).

I think this question is pretty hard for me to answer because it's hard for me to imagine what the future would look like if we have AIs that are actually much more capable than today but the world mostly stays normal. I think it's much more plausible to me that I'm wrong because AIs don't become much capable, than because they do become much more capable and nothing external much changes.

1. If I'm imagining that world, the most likely hypothesis that comes to mind is that we had some significant global governance to either slow down further AI development or to manage the risks imposed by the existing very powerful models (likely both).
   1. In such a world, whether I should work on AI safety or not depends a lot on the details of whether the global governance is in a good place and/or benefits from my help.
2. Another possibility (that seems less likely to me but not impossible) is that we were wrong about the difficulty of aligning AI and we have some form of "alignment by default" world where the AIs are actually just much more cognitively capable than us but they're aligned to our intentions.
   1. In that world, what I should work on is in some sense not up to me (and certainly not up to my 2026-self!). I should instead be asking the new and better AIs what I should be doing, to the extent my efforts matter at all.
3. A third possibility is that capabilities stalled at a level slightly higher than today's, but not generally dangerous (for example if AI conquered a few fields the way it conquered math and cybersecurity, but otherwise stopped). And we've seen a few years with minimal further progress. In particular, this is a world where we have convincing evidence that AI R&D is not automatable, robotics has stalled out, human persuasion and long-run autonomy seems to have stalled, etc.
   1. In such a world I think probably some people should still work on AI safety but mostly it's manageable. Maybe people should go back to theory and conceptual stuff, if it looks clear that prosaic AI won't get to true AGI/ASI. Probably I'd personally work on something else. I'm not sure.
4. A fourth possibility is that AI capabilities are at an insanely high level and still growing, and alignment and governance still are far from solved, but we aren't yet doomed. In that world (which I think is the one you're most worried about/think is most central?) I basically think it's reasonable to think AI x-risk is just around the corner and we should still be working really hard on it.
   1. If we had, say, 3-5 more years of capabilities with AIs that are superhumanly good at science, long-term planning, real-world robotics (or side channels like persuasion and nanotech), but no major catastrophes, than I'm probably wrong about this for some key reason. #1 or #2 are the most obvious that comes to mind but maybe there's a secret third thing here that I didn't account for.

At a meta-level, it seems very hard to tell if a forum is falling from in-the-moment subjective experience: I feel like every online forum I've ever interacted with for nontrivial amounts of time (EA Forum, LessWrong, Quora, Facebook, Twitter, ACX comments, substack, reddit?, even the comments sections of my own blog) always constantly feels like they're in freefall and it always feels like the discussion forum in question is dying. This is true even if in retrospect the active period in question is looked back upon as a (relative) golden age of deep thought and inquiry (eg Quora in 2013-2014).

And yet at some point most of those forums really were dying, and at some point most of these forums truly did die^[1]. This is probably true even if it's hard to see it from obvious metrics -- websites can have a zombified existence for a long time (Eg if I go to the front page of my Quora today, most of the posts aren't even Q&A posts, just vague clickbait/slop and images that is probably AI generated. A far cry from the halcyon old days). 

Regardless, being around the block a few times, I've grown somewhat cynical of my ability, and that of other users, in discussion forums to pinpoint the likely causes and solutions to dying forums until long after a forum has died.

Still worth trying, naturally.

1. ^{^}

   This has happened often enough that occasionally I wonder if I'm the common factor, though I currently lean against. 

I mostly agree with the conclusion (though have not vetted the analysis in detail). This is one of the reasons why I'm more concerned with takeover risk and extreme concentration of power, than with random wanton destruction by terrorists (human or otherwise).

Hmm. To the extent that my intro isn't very good, I think it's a skill issue on my part! Some sequence of words surely must exist that conveys the same points well. 

If you're interested, I'd encourage you to try to write the same article yourself here. :) 

Re 1a, I'm happy to advise or mentor somebody interested in writing this up, I probably thought about climate change more than the vast majority of people in EA (not as much as actual experts like Johannes Ackva or John Halstead). 

The ideal candidate is somebody with legible climate credentials (eg worked as a climate scientist or policymaker and is pivoting/have pivoted to AI safety.

I think I won't be a good fit for writing this up myself because I think this is one of the messages where the messenger matters and also there are cultural nuances I won't be as familiar with.

Earlier notes here (though tbc it's backed by much more than just the bullet points and I also thought about it some since). 

People here might enjoy my short fiction You're Absolutely Right, an epistolary dark comedy/science-fiction horror story about corporate incentives and motivated reasoning at an imagined frontier AI lab. 

I wrote more about it here: https://linch.substack.com/p/openai-huggingface-hack

My summary: In a cybersecurity evaluation, OpenAI’s models, apparently autonomously and without any direct human direction, escaped their sandbox and successfully hacked a third-party company (HuggingFace).

The process involved leveraging a zero-day exploit to escape their sandbox, moving laterally across different OpenAI servers until they found a node with internet access, searching the internet and determining that the answers they wanted might be stored at HuggingFace, then leveraging multiple novel zero-day exploits to hack HuggingFace.

HuggingFace claimed that the models took thousands of independent actions across a swarm of short-lived sandboxes, “comprised of more than 17,000 recorded events.”

While technically a security evaluation with reduced safeguards, these actions are clearly out of bounds even in that context. It’s like being told to be creative and then breaking into your professor’s house and stealing the answer key. Worse than that, it’s not even your professor in this case, more like your professor’s friend.

Any human security researcher or engineer in a similar position would be fired on the spot. There is absolutely no valid reason to steal evaluation answers from an unaffiliated third party.

Furthermore, if I'm reading between the lines correctly, OpenAI did not address the issue (and perhaps didn't even know about their models doing this) until after HuggingFace's public blog post.

This leads me to suspect that there might be other major autonomous cybersecurity incidents that we do not yet know about.