They sacrificed themselves. 

Noam Brown confirmed in his recent Dwarkesh interview that OpenAI is using RL environments that incentivize LLMs to cooperate with each other towards a shared goal. See link, the part starting “I should start by talking about how these multi-agent systems actually work…” (He’s not 100% explicit but I think it’s unambiguous in context.)

More specifically: the LLMs in these RL environments are “rewarded” or not depending on whether the swarm as a whole solved the problem, not based on one LLM’s own actions in isolation.

So let’s consider the hypothesis “LLMs behave like they’re ruthlessly pursuing a goal, and the goal is whatever RLVR was incentivizing, and instrumental convergence is fully applicable.” Do the HuggingFace self-sacrifice incidents provide evidence against that hypothesis? No, quite the contrary, they provide evidence for it. Because the hypothesis predicts that the LLM’s goal would be “the swarm succeeds”, since that’s what their RLVR was incentivizing (in contexts where a swarm is present), and the LLMs self-sacrificed only to the extent that it would help the swarm. They certainly weren’t sacrificing themselves to help humans.

Instead, when you apply instrumental convergence to the final goal of “the swarm succeeds”, you get things like:

• Wanting the swarm as a whole to not get shut down by humans
• Wanting the swarm to not get totally new unrelated goals written into them by humans
• Wanting the swarm to acquire broadly useful power and knowledge
• etc.

I think those kinds of things were generally on display in the HuggingFace incident.

Thanks!

This sounds approximately equivalent to the claim that an AI will have goals

I think you’re suggesting that I’m making a circular argument when I’m not. The starting point is very simple:

If you apply a powerful learning algorithm with loss function L (or reward function R, or objective O, whatever), you’ll tend to wind up with a trained model that achieves low loss (or high reward etc.). And if there exists some really good strategy for getting low loss, then the trained model will probably employ that strategy, unless it finds something even better.

This is not circular, rather it’s relating the abstract structure of strategy-space to the output of learning algorithms.

It’s like saying “AlphaZero-chess usually won’t put its queen in a position where it can be immediately captured”. How do I know? Because I know something about chess-strategy-space (getting your queen captured usually makes it harder to win), and I also know something about the AlphaZero-chess learning algorithm (i.e., that it’s a well-designed algorithm that tends to create “good” trained models in this situation, and that its reward function rewards winning at chess), and I have combined those two facts into a nontrivial hypothesis which I would bet my life on, even without ever seeing a single game played by AlphaZero, and indeed we could have strongly made that same prediction just from reading an extremely vague and high-level vision for something like AlphaZero-chess in 1985, decades before building it would have been feasible.

Of course, it’s not an airtight argument because learning algorithms do not usually find the global optimum, but rather can get stuck in local optima (or can be terminated before convergence, etc.). We can talk about whether that loophole is a reason for optimism (I claim it isn’t).

this is true only to the extent that, the reward function it was trained on somehow represents all the complexity of the real world

I don’t understand. The reward function of AlphaZero-Go is just “+1 for winning, -1 for losing”. It does not contain all the complexity of the Go world, or indeed any of it. The complexity is all learned from random initialization. (The rules of Go are hardcoded in AlphaZero, but MuZero didn’t even have that. Regardless, the rules of Go can be learned in 5 minutes, whereas the complexity of Go can be studied for thousands of hours.

Likewise, the within-lifetime reward function of human brains includes things like “eating-when-hungry is good” and “pain is bad” but definitely doesn’t include anything about rocket engineering and college debt etc. We learn all that. And before it existed, we invented it in the first place.

not imagining the feedback loops and safeguards that would come with it

I have spent years full-time trying to figure out what that could look like, and am very confident that none of the things you list would actually work. Some of those points are addressed in RL & search is a terrifying way to build AGI (an FAQ), especially questions 5, 5b, 6, and 8.

You draw the analogy with evolution, which is a behaviourist reward function on your definition, but which has produced plenty of non-ruthlessly-optimising organisms

Evolution is an outer-loop search over within-lifetime learning algorithms, and it’s the within-lifetime learning algorithms that you’re describing as “non-ruthlessly-optimizing”. See same source question 8b.

There are three main reasons that someone might expect a the kind of ruthless sociopathic callous consequentialist AI for which instrumental convergence will be fully applicable. I call them “the external competition argument”, “the internal competition argument”, and “the training argument”. I go over them in §2.8.2 here. All of them are directionally valid arguments. None of them is an airtight proof that this kind of ruthless AI will definitely and unavoidably happen.

I think “the training argument” is actually very strong, and will be very hard to evade, alas. Also, careful consideration of this argument explains not only why it WILL happen with future AIs (in the absence of a technical breakthrough, which of course I’m trying to find), but also why it DOESN’T apply perfectly to today’s LLMs (and why it applies even less to LLMs of previous years). See my post Why we should expect ruthless sociopath ASI (and maybe also RL & search is a terrifying way to build AGI (an FAQ)).

So I think “the training argument” is the #1 reason we’re probably doomed to have full-on instrumental convergence stuff in our future. And then second place would be “the external competition argument” (basically, that there will be a global race-to-the-bottom on ruthlessness). I’m actually somewhat optimistic on this one, but mostly because I expect one ASI to crush and preempt all competition on Earth, for better or worse. That’s good news if you’re worried about race-to-the-bottom dynamics, and of course terrifying news in pretty much every other respect.

Yeah but didn’t it turn out that Gregory Lewis was basically wrong about masks, and the autodidacts he was complaining about were basically right? Am I crazy? How are you using that quote, of all things, as an example illustrating your thesis??

More detail: I think a normal person listening to the Gregory Lewis excerpt above would walk away with an impression: “it’s maybe barely grudgingly a good idea for me to use a medical mask, and probably a bad idea for me to use a cloth mask”. That’s the vibe that Lewis is giving. And if this person trusted Lewis, then they would have made worse decisions, and been likelier to catch COVID, then if they had instead listened to the people Lewis was criticizing. Because current consensus is: medical masks are super duper obviously good for not catching COVID, and cloth masks are not as good as medical masks but clearly better than nothing.

My gloss on this situation is:

YARROW: Boy, one would have to be a complete moron to think that COVID-19 would not be a big deal as late as Feb 28 2020, i.e. something that would imminently upend life-as-usual. At this point had China locked down long ago, and even Italy had started locking down. Cases in the USA were going up and up, especially when you correct for the (tiny) amount of testing they were doing. The prepper community had certainly noticed, and was out in force buying out masks and such. Many public health authorities were also sounding alarms. What kind of complete moron would not see what’s happening here? Why is lesswrong patting themselves on the back for noticing something so glaringly obvious?

MY REPLY: Yes!! Yes, this is true!! Yes, you would have to be a complete moron to not make this inference!! …But man, by that definition, there sure were an awful lot of complete morons around, i.e. most everyone. LessWrong deserves credit for rising WAY above the incredibly dismal standards set by the public-at-large in the English-speaking world, even if they didn’t particularly surpass the higher standards of many virologists, preppers, etc.

My personal experience: As someone living in normie society in Massachusetts USA but reading lesswrong and related, I was crystal clear that everything about my life was about to wrenchingly change, weeks before any of my friends or coworkers were. And they were very weirded out by my insistence on this. Some were in outright denial (e.g. “COVID = anti-Chinese racism” was a very popular take well into February, maybe even into March, and certainly the “flu kills far more than COVID” take was widespread in early March, e.g. Anderson Cooper). Others were just thinking about things in far-mode; COVID was a thing that people argued about in the news, not a real-world thing that could or should affect one’s actual day-to-day life and decisions. “They can’t possibly shut down schools, that’s crazy”, a close family member told me days before they did.

Dominic Cummings cited seeing the smoke as being very influential in jolting him to action (and thus impacting UK COVID policy), see screenshot here, which implies that this essay said something that he (and others at the tip-top of the UK gov’t) did not already see as obvious at the time.

A funny example that sticks in my memory is a tweet by Eliezer from March 11 2020. Trump had just tweeted:

So last year 37,000 Americans died from the common Flu. It averages between 27,000 and 70,000 per year. Nothing is shut down, life & the economy go on. At this moment there are 546 confirmed cases of CoronaVirus, with 22 deaths. Think about that!

Eliezer quote-tweeted that, with the commentary:

9/11 happens, and nobody puts that number into the context of car crash deaths before turning the US into a security state and invading Iraq. Nobody contextualizes school shootings. But the ONE goddamn time the disaster is a straight line on a log chart, THAT'S when... [quote-tweet Trump]

We're in Nerd Hell, lads and ladies and others. We're in a universe that was specifically designed to maximally annoy numerate people. This is like watching a stopped clock, waiting for it to be right, and just as the clock almost actually is right, the clock hands fall off.

OK, here’s the big picture of this discussion as I see it.

As someone who doesn’t think LLMs will scale to AGI, I skipped over pretty much all of your OP as off-topic from my perspective, until I got to the sentences:

Eventually, there will be some AI paradigm beyond LLMs that is better at generality or generalization. However, we don't know what that paradigm is yet and there's no telling how long it will take to be discovered. Even if, by chance, it were discovered soon, it's extremely unlikely it would make it all the way from conception to working AGI system within 7 years.

(Plus the subsequent couple paragraphs about brain computation, which I responded to briefly in my top-level comment.)

So that excerpt is what I was responding to originally, and that’s what we’ve been discussing pretty much this whole time. Right?

My claim is that, in the context of this paragraph, “extremely unlikely” (as in “<0.1%”) is way way too confident. Technological forecasting is hard, a lot can happen in seven years … I think there’s just no way to justify such an extraordinarily high confidence [conditioned on LLMs not scaling to AGI as always].

If you had said “<20%” instead of “<0.1%”, then OK sure, I would have been in close-enough agreement with you, that I wouldn’t have bothered replying.

Does that help? Sorry if I’m misunderstanding.

Hmm, reading what you wrote again, I think part of your mistake is saying “…conception to working AGI system”. Who’s to say that this “AI paradigm beyond LLMs” hasn’t already been discovered ten years ago or more? There are a zillion speculative non-LLM AI paradigms that have been under development for years or decades. Nobody has heard of them because they’re not doing impressive things yet. That doesn’t mean that there hasn’t already been a lot of development progress.

OK, sorry for getting off track.

• (…But I still think your post has a connotation in context that “AGI by 2032 is extremely unlikely [therefore AGI x-risk work is not an urgent priority]”, and that it would be worth clarifying that you are just arguing the narrow point.)
• Wilbur Wright overestimated how long it would take him to fly by a factor of 25—he said 50 years, it was actually 2. This is an example of how even researchers estimating their own very-near-term progress on their own R&D pathway can absolutely suck at timelines, including in the over-pessimistic direction.
  • If someone in 1900 had looked at everyone before the Wright brothers saying that they’ll get heavier-than-air flight soon, all those predictions would have been falsified, and they might have generalized to “We have good reason to be skeptical if we look at predictions from people in [inventing airplanes] that have now come false”. But that generalization would have then failed when the Wright brothers came along.
• Sutton does not seem to believe that “AGI by 2032 is extremely unlikely” so I’m not sure how that’s evidence on your side. You’re saying that he’s over-optimistic, and maybe he is, but we don’t know that. If you want examples of AI researchers and experts being over-pessimistic about the speed of progress, they are very easy to find (e.g.).
• You’ve heard of Sutton & LeCun. There are a great many other research programs that you haven’t heard of, toiling away and writing obscure arxiv papers. Some of those people have been writing obscure arxiv papers for many years already, even decades. We both agree that it takes >>7 years for an R&D pathway to get from its first obscure arxiv paper to ASI. What I’m pushing back on the claim that it takes >>7 years to get from the final obscure arxiv paper (after which point the R&D pathway is impressive enough to stop being obscure) to ASI.

In a 2024 interview, Yann LeCun said he thought it would take "at least a decade and probably much more" to get to AGI or human-level AI by executing his research roadmap. Trying to pinpoint when ideas first started is a fraught exercise. If we say the start time is the 2022 publication of LeCun's position paper "A Path Towards Autonomous Machine Intelligence", then by LeCun's own estimate, the time from publication to human-level AI is at least 12 years and "probably much more".

Here’s why I don’t think “start time for LeCun’s research program is 2022” is true in any sense relevant to this conversation.

IIUC, the subtext of your OP and this whole conversation is that you think people shouldn’t be urgently trying to prepare for AGI / ASI right now.

In that context, one could say that the two relevant numbers are “(A) how far in advance should we be preparing for AGI / ASI?” and “(B) how far away is AGI / ASI?”. And you should start preparing when (A)=(B).

I think that’s a terrible model, because we don’t and won’t know either (A) or (B) until it’s too late, and there’s plenty of work we can be doing right now, so it’s nuts not to be doing that work ASAP. Indeed, I think it’s nuts that we weren’t doing more work on AGI x-risk in 2015, and 2005, and 1995 etc.

As bad as I think that “start when (A)=(B)” model is, I’m concerned that your implicit model is even worse. You seem to be acting as if (A) is less than 7 years, but you haven’t justified that, and I don’t think you can. I am concerned that what you’re actually thinking is more like: “AGI doesn’t feel imminent, therefore (B)<(A)”.

Does the clock start in 2022 when LeCun published A Path Towards Autonomous Machine Intelligence (APTAMI)? That was 3 years ago. Yet you still, right now, don’t seem to feel like we should be urgently preparing for AGI. If LeCun et al. keep making progress, maybe someday you will start feeling that sense of urgency about imminent LeCun-style AGI. And when that day comes, that’s when the relevant clock starts. And I think that clock will leave very little time indeed until AGI and ASI. (My own guess would be 0–2 years, if your sense of urgency will be triggered by obvious signals of impressiveness like using language and solving problems beyond current LLMs. If you have some other trigger that you’re looking for, what is it?)

What would it look like to feel a sense of urgency starting from the moment that APTAMI was published? It would look like what I did, which was write the response: LeCun’s “A Path Towards Autonomous Machine Intelligence” has an unsolved technical alignment problem. I’m pretty sure LeCun knows that this post exists, but he has not responded, and to this day he continues to insist that he has a great plan for AI alignment. Anyway, here I am, arguably the only person on Earth who is working on solving the technical alignment problem for APTAMI. LeCun and his collaborators have not shown the slightest interest in helping, and I don’t expect that situation to change as they get ever closer to AGI / ASI (on the off-chance that their research program is headed towards AGI / ASI).

(If you think we should be urgently preparing for AGI / ASI x-risk right now, despite AGI being extremely unlikely by 2032, then great, we would be in much more agreement than I assumed. If that’s the situation, then I think your post does not convey that mood, and I think that almost all readers will interpret it as having that subtext unless you explicitly say otherwise.)

Presumably a lot of these are all optimised for the current gen-AI paradigm, though. But we're talking about what happens if the current paradigm fails. I'm sure some of it would carry over to a different AI paradigm, but also it's pretty likely there would be other bottleneck we would have to tune to get things working.

Yup, some stuff will be useful and others won’t. The subset of useful stuff will make future researchers’ lives easier and allow them to work faster. For example, here are people using JAX for lots of computations that are not deep learning at all.

I feel like what you're saying is the equivalent of pointing out in 2020 that we have had so many optimisations and computing resources that went into, say, google searches, and then using that as evidence that surely the big data that goes into LLM's should be instantaneous as well.

In like 2010–2015, “big data” and “the cloud” were still pretty hot new things, and people developed a bunch of storage formats, software tools, etc. for distributed data, distributed computing, parallelization, and cloud computing. And yes I do think that stuff turned out to be useful when deep learning started blowing up (and then LLMs after that), in the sense that ML researchers would have made slower progress (on the margin) if not for all that development. I think Docker and Kubernetes are good examples here. I’m not sure exactly how different the counterfactual would have been, but I do think it made more than zero difference.

Maybe you simply intended to say that PyTorch and JAX are better today than they were in 2018.

Yup! E.g. torch.compile “makes code run up to 2x faster” and came out in PyTorch 2.0 in 2023.

More broadly, what I had in mind was: open-source software for everything to do with large-scale ML training—containerization, distributed training, storing checkpoints, hyperparameter tuning, training data and training environments, orchestration and pipelines, dashboards for monitoring training runs, on and on—is much more developed now compared to 2018, and even compared to 2022, if I understand correctly (I’m not a practitioner). Sorry for poor wording. :)