A modest update on this apparently now notorious table: 1,370 days later, while it has been turned off for various periods AFAICT, I still routinely pass through Bayes and observe groups at conferences/events gathered around it and staring at it, and it routinely catches the eye of visitors. No one ever stares at or mentions any of the other tables, $300 or otherwise.

FWIW, I am skeptical of the interpretation you put on that graph. I considered the same issue in my original analysis, but went with Strenze 2007's take that there doesn't seem to be much steeper IQ/income slope and kept returns constant. Seb objects that the datapoints have various problems and that's why he redoes an analysis with NLS to get his number, which is reasonable for his use, but he specifically disclaims projecting the slope out (which would make ES more profitable than it seems) like you do:

If the estimate were only based on the more recent cohort (NLSY97), the estimated effect between the ages of 25 and 65 would be $74,508, and adding the rough estimates for the additional years (18-24 and 66-80) increases it to $90,208. I think this estimate is too high; this projection relies on the assumption that the increases in the mean and variance of wages will hold for the future, which may not be the case.

And it's not hard to see why the slope difference might be real but irrelevant: people in NLSY79 have different life cycles than in NLSY97, and you would expect a steeper slope (but same final total lifetime income / IQ correlation) from simply lengthening years of education - you don't have much income as a college student! The longer you spend in college, the higher the slope has to be when you finally get out and can start your remunerative career. (This is similar to those charts which show that Millennials are tragically impoverished and doing far worse than their Boomer parents... as long as you cut off the chart at the right time and don't extend it to the present.) I'm also not aware of any dramatic surge in the college degree wage-premium, which is hard to reconcile with the idea that IQ points are becoming drastically more valuable.

So quite aside from any Inside View takes about how the current AI trends look like they're going to savage holders of mere IQ, your slope take is on shaky, easily confounded grounds, while the Outside View is that for a century or more, the returns to IQ have been remarkably constant and have not suddenly been increasing by many %.

Did it even occur to you that saying some things in some countries is illegal and your doxxing victim could go to prison for saying something that looks innocuous to you? Do you know where Mark is from and what all his country’s speech laws are?

If Mark is in such a situation (which he was not, and I knew he was not), then the real culprit is whoever implemented such a completely broken and utterly unfixable 'anonymous' comment, and himself for being a security researcher and yet believing that retroactively making comments 'anonymous' on a publicly-scrape-able website would protect him against nation-state actors when anonymity was neither the goal nor a documented promise of the account deletion feature he was abusing and then crying 'dox!' about it not doing what it wasn't supposed to do and didn't do.

I don't think it's odd at all. As the Bloomberg article notes, this was in response to the martyrdom of Saint Altman, when everyone thought the evil Effective Altruists were stealing/smashing OA for no reason and destroying humanity's future (as well as the fortune of many of the individuals involved, to a degree few of them bothered to disclose) and/or turning it over to the Chinese commies. An internal memo decrying 'Effective Altruism' was far from the most extreme response at the time; but I doubt Gomez would write it today, if only because so much news has come out since then and it no longer looks like such a simple morality play. (For example, a lot of SV people were shocked to learn he had been fired from YC by Paul Graham et al for the same reason. That was a very closely held secret.)

Good report overall on tacit knowledge & biowarfare. This is relevant to the discussion over LLM risks: the Aum Shinrikyo chemist could make a lot of progress by reading papers and figuring out his problems as he went, but the bacteriologist couldn't figure out his issues for what seems like what had been a viable plan to weaponize & mass-produce anthrax but where lack of feedback led it to fail. Which does sound like something that a superhumanly-knowledgeable (but not necessarily that intelligent) LLM could help a lot with simply by pattern-matching and making lists of suggestions for things that are to the human 'unknown unknowns'.

If a crazy person wants to destroy the world with an AI-created bioweapon

Or, more concretely, nuclear weapons. Leaving aside regular full-scale nuclear war (which is censored from the graph for obvious reasons), this sort of graph will never show you something like Edward Teller's "backyard bomb", or a salted bomb. (Or any of the many other nuclear weapon concepts which never got developed, or were curtailed very early in deployment like neutron bombs, for historically-contingent reasons.)

There is, as far as I am aware, no serious scientific doubt that they are technically feasible: multi-gigaton bombs could be built or that salted bombs in relatively small quantities would render the earth uninhabitable to a substantial degree, for what are also modest expenditures as a percentage of GDP etc. It is just that there is no practical use of these weapons by normal, non-insane people. There is no use in setting an entire continent on fire, or in long-term radioactive poisoning of the same earth on which you presumably intend to live afterwards.

But you would be greatly mistaken if you concluded from historical data that these were impossible because there is nothing in the observed distribution anywhere close to those fatality rates.

(You can't even make an argument from an Outside View of the sort that 'there have been billions of humans and none have done this yet', because nuclear bombs are still so historically new, and only a few nuclear powers were even in a position to consider whether to pursue these weapons or not - you don't have k = billions, you have k < 10, maybe. And the fact that several of those pursued weapons like neutron bombs as far as they did, and that we know about so many concepts, is not encouraging.)

That was covered in my 2023-11-25 comment.

Hm, maybe it was common knowledge in some areas? I just always took him for being concerned. There's not really any contradiction between being excited about your short-term work and worried about long-term risks. Fooling yourself about your current idea is an important skill for a researcher. (You ever hear the joke about Geoff Hinton? He suddenly solves how the brain works, at long last, and euphorically tells his daughter; she replies: "Oh Dad - not again!")

Ilya has always been a doomer AFAICT, he was just loyal to Altman personally, who recruited him to OA. (I can tell you that when I spent a few hours chatting with him in... 2017 or something? a very long time ago, anyway - I don't remember him dismissing the dangers or being pollyannaish.) 'Superalignment' didn't come out of nowhere or surprise anyone about Ilya being in charge. Elon was... not loyal to Altman but appeared content to largely leave oversight of OA to Altman until he had one of his characteristic mood changes, got frustrated and tried to take over. In any case, he surely counts as a doomer by the time Zilis is being added to the board as his proxy. D'Angelo likewise seems to have consistently, in his few public quotes, been concerned about the danger.

A lot of people have indeed moved towards the 'doomer' pole but much of that has been timelines: AI doom in 2060 looks and feels a lot different from AI doom in 2027.

1. I haven't seen any coverage of the double structure or Anthropic exit which suggests that Amodei helped think up or write the double structure. Certainly, the language they use around the Anthropic public benefit corporation indicates they all think, at least post-exit, that the OA double structure was a terrible idea (eg. see the end of this article).

2. You don't know that. They seem to have often had near majorities, rather than being a token 1 or 2 board members.

   By most standards, Karnofsky and Sutskever are 'doomers', and Zillis is likely a 'doomer' too as that is the whole premise of Neuralink and she was a Musk representative (which is why she was pushed out after Musk turned on OA publicly and began active hostilities like breaking contracts with OA). Hoffman's views are hard to characterize, but he doesn't seem to clearly come down as an anti-doomer or to be an Altman loyalist. (Which would be a good enough reason for Altman to push him out; and for a charismatic leader, neutralizing a co-founder is always useful, for the same reason no one would sell life insurance to an Old Bolshevik in Stalinist Russia.)

   If I look at the best timeline of the board composition I've seen thus far, at a number of times post-2018, it looks like there was a 'near majority' or even outright majority. For example, 2020-12-31 has either a tie or an outright majority for either side depending on how you assume Sutskever & Hoffman (Sutskever?/Zilis/Karnofsky/D'Angelo/McCauley vs Hoffman? vs Altman/Brockman), and with the 2021-12-31 list the Altman faction needs to pick up every possible vote to match the existing 5 'EA' faction (Zilis/Karnofsky/D'Angelo/McCauley/Toner vs Hurd?/Sutskever?/Hoffman? vs Brockman/Altman) although this has to be wrong because the board maxes out at 7 according to the bylaws so it's unclear how exactly the plausible majorities evolved over time.