Thanks for asking! It seems good to reflect on this.

I previously mentioned that I had considered titling this "EA in a World Where People Actually Listen to Us" and I think that would have been a more accurate title.

I think I was reasonably correct to point to the FLI letter as a turning point in AIS becoming mainstream. 

I was not that correct to point to lack of funding from Meta/FTX being a turning point, as a lot of people who got rich from AI are now donating to EA causes.

None of my three possible futures seem to have really occurred. "Politics and Civil Society" feels closest, as EA (particularly AIS) work increasingly involves working with the public, but neither of these descriptions is accurate: "There is substantial public appetite to heavily regulate AI. The technical challenges end up being relatively easy." 

If I had to name my current vibe, it would be something like "EA as a signal of depth of commitment, rather than stated beliefs." There are now broad swathes of society where "AI is dangerous" is taken as a given, but there are still very few people who change their careers, donate, or take other costly steps. I expect to continue to see that the people who make headlines as whistleblowers, donate large fractions of their incomes, or otherwise take costly steps to continue to be disproportionately people who know what an insecticide-treated bednet is.^[1]

1. ^{^}

   Although I think something of the opposite has happened in global development, where fewer people now give lip service to foreign aid. Animal welfare continues to be "almost everyone says animal cruelty is bad, almost no one does anything about it."

That archetype has largely evaporated. These people are now 30-something

I agree that those specific people are now 30-something, but I still see a lot of this archetype on twitter, substack, etc. 

You can just do things

My experience is that people often don't post for the mundane reason that they didn't really think about it.^[1] Contests are a good way of counteracting this, but the monetary reward is usually not very motivating, it's just that it makes "posting about X" become a Thing with a Deadline etc.

An implication of this is that anyone (even you, dear reader) can solve this problem with a couple days of work:

1. Message a few people that authors would value feedback from, say "hey, if I got people to write about X, would you commit to replying?"
2. Message a few prospective authors, say "hey, I've gotten these people to commit to giving feedback, would you be willing to post something?"
3. Once you've got 5-10 people to agree, post that it's happening
4. Follow up a few times with reminders because people are disorganized

This is basically all I did for the AI Pause Debate, for example, and that seemed to go fairly well. 

1. ^{^}

   For example: I am commenting here because someone messaged me to suggest that I do so. Otherwise I likely wouldn't have seen this post.

Thanks for writing this, I thought it was interesting.

I do think that early work was making a claim more like "reward maximization will lead to IC" (which, as you link to, is provably true in certain setups), and it's worth re-evaluating those claims in light of the fact that a lot of modern ML doesn't look like "reward the model for maximizing a fixed objective."

One prediction which would come out of this is that we should see more instrumental convergence if/when RL becomes a larger part of the training picture.

Isn't the answer to "At what level of capability is this mythical hard left turn towards meglomania meant to suddenly show up simultaneously in all these AI's" simply "when megalomania is instrumentally useful?"

It seems to me that when I, e.g., ask an agent to add a button to my website, it would not be more likely to be successful if it first tried to overthrow the US government. So the fact that it doesn't try to overthrow the US government doesn't seem to give much evidence either way about IC.

It seems to me that negative PR of EA over time has correlated quite well with how influential we were at that time.^[1] "People hate philosophers" doesn't explain why the hate focuses on EA in particular:

I'm in favor of reiterating the points you make about our track record, but also we should be realistic that "I support [policy] but hate those extremists who advocate for [policy]" is what success looks like. E.g. MLK was extremely unpopular^[2] whilst civil rights legislation was being passed: 

1. ^{^}

   And also how generally newsworthy we were. One of the clearest learnings I have about EA comms is that a story will get >10x the traction if the journalist can somehow include "Elon Musk" in the title.

2. ^{^}

   As comparison: LBJ, a president so unpopular he didn't even attempt to run for re-election, bottomed out at a net approval 13 points higher than MLK

Thanks for starting this group!

Thanks Dan, I was about to tell you that I meant that the null hypothesis can't be rejected,^[1] but then I asked Opus to confirm. Opus says that, while all of the metrics I posted here do indeed show a nonsignificant difference, age-adjusted post-chatGPT YC companies are worth 1.5x more on a log scale, and that this is significant at p = 0.001 (which survives corrections for multiple comparisons).

I've changed the title to say that they aren't growing "much" faster, which hopefully conveys the accurate takeaway.

1. ^{^}

   I tried to say this informally in the article: "YC companies are arguably growing a bit faster than pre-2023 companies (including AfterQuery, YC’s fastest-ever unicorn), but the difference isn’t large relative to the underlying variance"

Thanks for writing this Geoffrey, it seems important.

it looks like most AI safety orgs don't have a mass communications function

I think this is true, with some exceptions (e.g. MIRI)