This is a special post for quick takes by Roman Ross. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Very low confidence in this take, but I suspect that in university groups, Better Futures fieldbuilding looks much better under short timelines than technical safety research fieldbuilding. As far as I can tell, the barriers to doing important stuff in that field are much lower.
It doesn't feel that way to me but I might not be aware of all the relevant considerations, for example what jobs or other paths you're thinking of for university groups to be involved. Can you elaborate a little?
Still low confidence, but after thinking on this more, I think the best case for this is in general advocacy to help protect the rights of digital minds (although it depends on how charismatic you are). I think most of the remaining tractability that a new EA could get into with a <2 year timeline is in various work that might shift the law (either comms or advocacy), and this seems the least likely to get immediately solved by a superintelligence (unlike i.e. space governance) while still having a major impact on the majority of sentient creatures that could exist
It seems like many people have been confused lately about why 1) AI safety orgs are in desperate need of good hires and 2) org hiring rates are very low. My understanding is that there are many reasons for this, but a 2021 post states it pretty well:
"A lot of EA content is subtle and detailed, and easy to accidentally compress into something misleading. For example, 80k might write a nuanced article saying 'You should focus on talent gaps, not funding gaps', but this gets translated into 'EA is talent constrained', and then people repeat that phrase without linking to the article, and then many people form an inaccurate belief that EA needs 'pretty talented people', rather than 'EA needs very specific talents that are missing.'"
The best career advice seems to be to focus on developing a rare and valuable skill that orgs currently need. In AI safety, this skill is a combination of specific context, agency, mission-alignment, and strategic vision. You can develop it by reading a lot, writing out your thoughts, talking to people at events, and completing real projects.
It's hard to do really impactful work. You need a plausible explanation for why your efforts are the best way your salary can be spent. This takes time, effort, and careful strategizing. Most of the world has high schools, colleges, and internships to train people for jobs, but a similar system is only recently becoming legible for AI safety and EA. The rare skills are rare for a reason. Still, I think the broader EA community could do a better job of communicating what it means for these "talent-gaps" to exist.
I'm very interested if anyone makes progress on how to compactly describe what the shape of the talent needed is. My current best attempt is both too abstract and I don't quite have the right words, so it ends up that I have a pile of slightly conflicting words to point at concept clusters that seem to me to be the main joints of the problem:
1. {understands/will-take-action-on/agency-directed-towards/internal-locus-of-control-about/curiosity-about} the core difficulties of 2. {taking-actions-now/intervening-towards/coming-up-with-plans-and-techniques-for} achieving worlds that 3. {become-good-fast-and-stay-good-until-2200-and-beyond/robustness-of/durability-of/ongoing-maintenance-of} 4. of {good-things-by-human-lights/morality/omnibenevolence/meaningfulness/humans-still-being-loadbearing/eigenmorality/omnispecies-utopia/goddess-of-everything-else/solve-moral-philosophy/extract-reference-to-good-from-humans-reliably} 5. in the face of {world significantly ruled by things like dark-triad/defect-leaning/international-Realism, competition/evolution/malthusian/goddess-of-cancer/nick_land::pythia/scott_alexander::moloch, force, machines/spreadsheets/algorithms/ai/robots}
Which is at least short, but is neither particularly readable nor particularly clear. Hopefully clear enough that someone who gets curious about that pile of keywords would end up a lot closer than "AI scary, maybe I pick first job off suggestions?" does.
Specific context, agency, mission-alignment, and strategic vision are all subjective and rather difficult to pin down.
‘You need a plausible explanation for why your efforts are the best way your salary can be spent’ - yes, if you're pitching to have a role created for you that isn't being advertised. Otherwise, the explanation is 'because I meet the criteria you are asking for’. Presumably, by the time the role is being advertised, the org has already decided that there’s work that needs doing, and that paying someone to do it is the best use of the money they have.
Many ads seem to say things to the effect of 'don't worry if you have no technical skills or relevant experience - we're just looking for good people, so if you don't meet all of our criteria, apply anyway. If you're not sure, err on the side of applying!' This seems like a waste of many people’s time.
I'd agree that the EA/AIS community could do a better job of communicating what it means by "talent gaps." Further, I think the feedback loops on failed attempts to secure a role are beyond poor (mostly just pointing to upskilling programs/research fellowships). While I get that orgs already have limited bandwidth for admissions, I think the feedback loop pipeline is a piece of this puzzle that should be improved.
Quick math: In terms of total expected suffering averted, inviting a group of friends to a barbecue where the main meat is beef is probably more impactful than eating a vegan meal by yourself.
According to Faunalytics' Animal Product Impact Scales, a serving of chicken requires approximately 10 times as many days of animal life to produce as a serving of beef (not adjusted for quality of life or moral weights). Additionally, most seafood is much, much worse than this. If this meal prevents your friends from going and eating those other foods, tricking them into eating beef will likely do more good than eating a vegan meal by yourself.
Still, this may not be as impactful as saving your money and donating it directly to impactful charities. Saving a single dollar on food and donating it to the SWP will probably do more to reduce suffering than hosting a barbecue.
Also, you might expose yourself to various forms of value drift by hosting a barbecue, but I could see this happening in a net positive (hosting more barbecues in the future) or a net negative (giving up on veganism entirely/permanently failing to convince any of your friends to give up meat) direction.
At my particular place, I think that improving AI safety community epistemics (writing thorough and careful arguments against people both in the community and outside of it who make silly mistakes) is, by most reasonable estimates, probably less impactful than helping to run MAIA, which is what I currently spend most of my time doing.
I don’t think I’m the only person in this situation, which is quite scary to me, as I think there are a lot of things are epistemically iffy in the AI safety community. For example:
A lot of the most viral pro-AIS people (especially the ones that are relatively new to the field) seem to not be reasoning especially carefully
The field’s hiring process is quite insular
Not enough time is spent engaging with a lot of good faith disagreements (imo)
Fieldbuilding strategy doesn’t optimize for careful reasoning amongst those it recruits
People receive lots of social/material benefits for continuing to believe that AI risk is the most important thing and work on it
Even under some of the shortest timelines and most speculative capabilities that superintelligence might have, I really don’t extinction within the next year should be assigned any notable probability mass, due to the need for data centers to be replenished
I also see a lot of disagreements that I think are poorly reasoned. For example:
Poor application of/thinking about EV when conceptualizing risks
Psychoanalysis that doesn’t explain the safety community’s behaviors well
Comments significantly overstating the case for something
Arguments without the slightest amount of research into what the other side actually believes
Tbh though, most of these critiques apply to both sides.
I think this primarily arises because the people working in AI safety who are the most careful reasoners also have the highest opportunity costs, because they tend to be good at other things. This means it’s less valuable for them to write out their thoughts online carefully.
My main point, though, is that it’s a little scary to see so much going wrong with the discourse and not being able to do anything about it (or even be worth trying to do anything about it). It feels like there’s not much worth trying except praying for Forethought to do a good job and reach everyone.
Very low confidence in this take, but I suspect that in university groups, Better Futures fieldbuilding looks much better under short timelines than technical safety research fieldbuilding. As far as I can tell, the barriers to doing important stuff in that field are much lower.
It doesn't feel that way to me but I might not be aware of all the relevant considerations, for example what jobs or other paths you're thinking of for university groups to be involved. Can you elaborate a little?
Still low confidence, but after thinking on this more, I think the best case for this is in general advocacy to help protect the rights of digital minds (although it depends on how charismatic you are). I think most of the remaining tractability that a new EA could get into with a <2 year timeline is in various work that might shift the law (either comms or advocacy), and this seems the least likely to get immediately solved by a superintelligence (unlike i.e. space governance) while still having a major impact on the majority of sentient creatures that could exist
Aimed at an audience newer to EA/AI safety
It seems like many people have been confused lately about why 1) AI safety orgs are in desperate need of good hires and 2) org hiring rates are very low. My understanding is that there are many reasons for this, but a 2021 post states it pretty well:
The best career advice seems to be to focus on developing a rare and valuable skill that orgs currently need. In AI safety, this skill is a combination of specific context, agency, mission-alignment, and strategic vision. You can develop it by reading a lot, writing out your thoughts, talking to people at events, and completing real projects.
It's hard to do really impactful work. You need a plausible explanation for why your efforts are the best way your salary can be spent. This takes time, effort, and careful strategizing. Most of the world has high schools, colleges, and internships to train people for jobs, but a similar system is only recently becoming legible for AI safety and EA. The rare skills are rare for a reason. Still, I think the broader EA community could do a better job of communicating what it means for these "talent-gaps" to exist.
I'm very interested if anyone makes progress on how to compactly describe what the shape of the talent needed is. My current best attempt is both too abstract and I don't quite have the right words, so it ends up that I have a pile of slightly conflicting words to point at concept clusters that seem to me to be the main joints of the problem:
Which is at least short, but is neither particularly readable nor particularly clear. Hopefully clear enough that someone who gets curious about that pile of keywords would end up a lot closer than "AI scary, maybe I pick first job off suggestions?" does.
Specific context, agency, mission-alignment, and strategic vision are all subjective and rather difficult to pin down.
‘You need a plausible explanation for why your efforts are the best way your salary can be spent’ - yes, if you're pitching to have a role created for you that isn't being advertised. Otherwise, the explanation is 'because I meet the criteria you are asking for’. Presumably, by the time the role is being advertised, the org has already decided that there’s work that needs doing, and that paying someone to do it is the best use of the money they have.
Many ads seem to say things to the effect of 'don't worry if you have no technical skills or relevant experience - we're just looking for good people, so if you don't meet all of our criteria, apply anyway. If you're not sure, err on the side of applying!' This seems like a waste of many people’s time.
I'd agree that the EA/AIS community could do a better job of communicating what it means by "talent gaps." Further, I think the feedback loops on failed attempts to secure a role are beyond poor (mostly just pointing to upskilling programs/research fellowships). While I get that orgs already have limited bandwidth for admissions, I think the feedback loop pipeline is a piece of this puzzle that should be improved.
Quick math: In terms of total expected suffering averted, inviting a group of friends to a barbecue where the main meat is beef is probably more impactful than eating a vegan meal by yourself.
According to Faunalytics' Animal Product Impact Scales, a serving of chicken requires approximately 10 times as many days of animal life to produce as a serving of beef (not adjusted for quality of life or moral weights). Additionally, most seafood is much, much worse than this. If this meal prevents your friends from going and eating those other foods, tricking them into eating beef will likely do more good than eating a vegan meal by yourself.
Still, this may not be as impactful as saving your money and donating it directly to impactful charities. Saving a single dollar on food and donating it to the SWP will probably do more to reduce suffering than hosting a barbecue.
Also, you might expose yourself to various forms of value drift by hosting a barbecue, but I could see this happening in a net positive (hosting more barbecues in the future) or a net negative (giving up on veganism entirely/permanently failing to convince any of your friends to give up meat) direction.
At my particular place, I think that improving AI safety community epistemics (writing thorough and careful arguments against people both in the community and outside of it who make silly mistakes) is, by most reasonable estimates, probably less impactful than helping to run MAIA, which is what I currently spend most of my time doing.
I don’t think I’m the only person in this situation, which is quite scary to me, as I think there are a lot of things are epistemically iffy in the AI safety community. For example:
I also see a lot of disagreements that I think are poorly reasoned. For example:
Tbh though, most of these critiques apply to both sides.
I think this primarily arises because the people working in AI safety who are the most careful reasoners also have the highest opportunity costs, because they tend to be good at other things. This means it’s less valuable for them to write out their thoughts online carefully.
My main point, though, is that it’s a little scary to see so much going wrong with the discourse and not being able to do anything about it (or even be worth trying to do anything about it). It feels like there’s not much worth trying except praying for Forethought to do a good job and reach everyone.