TLDR. AI Safety is bottlenecked by high-context senior talent, not talent. We should communicate this more clearly to manage the expectations of those looking to break into the field.
Cross-posted on Substack.
This post was inspired by my friend Jen’s quick take. I’m repeating some of her points, but I wanted to add my 2 cents. I’m also going to toot my own horn and refer you to a previous post of mine about the importance of context and how to build it. I recommend reading that first if you haven’t already.
The Bottleneck Paradox
We have been hearing a lot about the talent bottleneck in AI Safety. In some ways this is true, but in others it’s misleading and leaves many people disillusioned and doubting whether they have anything to contribute to AI Safety.
Meanwhile, the field is becoming harder to penetrate. Some reasons for this are:
- Organisations are better funded than they were a few years ago, which means they are less keen on working with volunteers and just want to hire full-time staff.
- Many of the “oldtimers” who could give the best career advice are now quite senior and oversee a larger number of people, which significantly raises the bar for talking to newcomers.
- The field has grown a lot, and many roles require more specialised AIS knowledge. There is more to read, and if you are new, it’s not easy to figure out which sub-area you should focus on.
Hell, I run into people with pretty impressive backgrounds, but they can't even get into a BlueDot course.
Subscribe to Building CapacitySo what’s going on?
Organisations are desperate to hire, but mainly for roles that require both many years of work experience and the high levels of context, which you build by having been around for years in the space. Here is a comment from Tom, who is a senior advisor at Probably Good.
A slight qualifier here is that getting to the level of context required for some jobs - especially senior ones that experienced professionals might be applying to - can take (sometimes much) longer, so it's important to have realistic expectations there. For instance, if you want to work in AI safety, and have a background (e.g. quantitative finance, venture capital) that could give you great skills to be a grantmaker, you'll likely still need to know more than just the high-level concepts and the landscape of organisations working on it; you might need to know the strengths and weaknesses of different theories of change, and have a sense of the wider funding landscape.
The people who can fill those roles are few and far between, and usually have already been working in AI Safety for years. If they become available, the headhunters are on their tails.
Some roles don’t need as much context, but:
If there are more openings (e.g. operations[1]), they get a large number of skilled applicants
- If they require a more niche background (e.g. marketing), then there aren’t as many opportunities available
Conclusion
Instead of saying AIS is bottlenecked by talent, I would like us to offer a more nuanced explanation, which looks something like this:
Some AI Safety roles demand a specific skill set, deep context, and years of work experience, which makes them hard to fill even when funding and talent aren’t constraints. Other roles are much more competitive, drawing many applicants since they don't require candidates to already have been around in AI Safety for a long time.
See also: Good Senior People Exist and We’re Wasting Them
- ^
Though it’s worth noting that orgs seem to want high-context people for senior ops roles too, whether or not this is actually needed.
At this point, it feels like the bottleneck being "talent" vs. "context" or anything else is just semantics.
Talent that has years of experience but doesn't have the specific knowledge I want is not the talent I need. I need a different talent. Therefore, it's still a talent bottleneck.
Maybe it would be better if AI safety orgs could be extremely more exact about what talent means to them, but that also leaves the field fairly open to a type of gaming a lot of people seem to want to avoid.
The field does act like a false positive hire is immensely worse than a false negative, though, which might not be the case.
I don't have a point, these are just random observations.
I agree that what you describe is technically still a talent constraint, but this nuance is mainly lost in translation, which is why we have this issue.
Terms of reference should say what experience and competencies are needed for a role. I dont see a downside with being more precise about this - ie these are mandatory things we're looking for, and these are desirable competencies. The more general a job advertisement is, the more candidates will apply, which creates unnecessary workload for the hiring organisation, and wastes the time of candidates who realistically were never competitive.
Agreed. There's a real aversion to hiring 'the wrong person'. But if you hire enough people, you'll eventually make a bad hire. Hiring / onboarding / probation / performance management should be seen as an ongoing process to get the best candidate and make sure they're working effectively, rather than relying 100% on the hiring stage.
As I've mentioned above, there are opportunity costs when leaving a position vacant, or hiring a more trusted but less experienced candidate and hoping they will 'grow into the role'. It seems like these costs are frequently underweighted.
Mmmm I think orgs should probably just be more honest with themselves as well - have the very specific criteria, sure, but just be honest about them. Like: if you're really only going to hire a unicorn, then just say unicorns only - that seems fine to me? And then we can just move on to arguing about the object-level question of do you actually need a unicorn.
I think one of the challenges as organisations scale rapidly is that organisations are hiring for positions that they'd never had before - eg they are hiring for a Head of talent, Head of Finance, Head of Operations. Since they've not had those roles before, there are two issues:
It's hard to know what the role should do, and related to this, what are the mandatory and desirable requirements.
One of the specific items that seems to be attracting some heat is the "if you're not sure this is for you, apply anyway" type lines on applications. That seems like a cheap one to fix - my interpretation of this one has historically been about credentialling: 'if you don't have the credential you think you might need, you might be surprised'. So maybe we could just come up with a new boilerplate paragraph that better communicates what is actually meant here.
Nice post!
Here's some reactions:
Context:
I think there's two things that make candidates more attractive that genuinely do take time to develop:
Talent bottleneck:
I think it's incorrect to say that there's a shortage of talented people: by definition, there is a shortage. E.g. We often joke that the best thing for AI safety would be a cloning machine to clone the best people.
The talent shortage is particularly relevant when you account for the fact that a majority of roles are only available in two locations: London and the Bay Area. This excludes a lot of candidates.
There's also also the prevailing myth that a great CV equates to great performance. Here's a meme that characterises my experience (and I think that of many others): https://drive.google.com/file/d/1sOdAeGFaiIHvdn3npElSsHeX07B3fcQm/view?usp=sharing
As someone who is currently hiring (and to put this bluntly), it can be difficult to find candidates who can follow instructions and/or produce a document that wasn't written by Claude. The bar is not high and I am somewhat confused about how the economy functions at this point.
Particularly for the first 3-5 hires, where you need to have a very high bar for talent in order to set the right trajectory for the organisation ... it's just not easy.
I think we’re mostly using “talent” differently here. If it means the whole package such as skills, experience, context, networks and being in the right location: then yes, there’s a shortage of people who tick all those boxes. My point is that “AI safety is talent-constrained” can sound to newcomers like broadly capable people are in short supply. I also don't envy you on the hiring side, though with all the AI-applications one gets these days:)
Thanks Gergo, this and your previous post / guide on building context are both really valuable.
I agree with your Venn diagram - there's plenty of very talented and experienced people around, with varying levels of context. Developing parts of that context is possible to do autonomously - eg following the steps in your earlier guide. But developing the networks, the sense of the field (who's doing what / where) is not easy, especially for people who are not located in a hub.
I agree that hiring organisations seem to want high-context for most roles, including operations roles, whether or not this is needed. It seems like this limits your pool of potential candidates - if you want a rock star head of finance, who also has high context, who is also available to work from your location or be hired from your location. When organisations cant find these unicorns, they're either leaving the roles unfilled, or hiring a person with less experience but 'potential to grow into the role'. I've previously commented that there's an opportunity cost to both of these options.
I've been on both sides of recruiting and job-seeking, so I know it's not easy for anyone. I would like to see more programmes like the Successif organisation builder's programme, which provide a way for experienced people to build context through paid part-time work, while also helping organisations that are trying to scale and deliver.
Great point, and I agree that there should be more orgs such as SuccessIf!
Kairos has $9.5 million for university groups alone. Most other organisations prioritise early stage careers. For senior talent to gain context, the same kind of money needs to be diverted. Currently the expectation is they must leave a stable income for over a year without any guarantee of success while the opportunities are diminished because the pathways prioritise young talent (and are run by young talent).
Context ought to be defined in terms of clear knowledge requirements. Things that can be documented, taught and assessed if needed. I expect much of what someone needs to know to do their job well could probably be learned in anywhere from a few hours to a few weeks of focused onboarding.
If that's not true, and orgs really do need people with years of movement-specific experience, they should say so. Instead, there’s a huge ecosystem of career advice, fellowships, introductory programmes and recruitment campaigns encouraging outsiders to consider AI safety careers.
I think these programmes are creating expectations that the actual hiring market cannot satisfy. Better to leave it to the headhunters if you ask me, and re-allocate the funding behind those programmes to GHD. Hard to justify why another talent-sourcing agency for AI safety is needed, from a cost effectiveness standpoint, when there are still bednets need distributing.
'Context' seems to be the latest iteration of 'culture fit' or 'mission alignment'. Another nebulous term that EA orgs use to explain why they keep reeling in hundreds, sometimes thousands, of applicants who basically don’t stand a chance.
The burden of proof should be on orgs invoking context as a hiring requirement, not on applicants to demonstrate possession of that indefinable, je ne sais quoi quality of context.
Strong disagree. A language is a way of life, not a set of memorised answers. I agree that it would be nice if there was some more honesty about what exactly talent-constrained means. I like how this post clarifies it.
Agree, there is a mismatch in communication between what AIS organizations actually need and what they are telling candidates. And although I consider necessary the talent infrastructure AIS is building, it is quite inefficient and problematic as I discussed here.
The main concern I see: fellowships and most training programs are focused on "low-hanging fruits". They are advertised as ways to bring experienced talent into the field, but rather they tend to select people who either would have less problems to enter a job, or people who were already in the field (I analysed the fellows backgrounds of famous fellowships such as GovAI or MATS and I can confidently say this is the case).
And if we are focused on just draining the talent pool of already best-suitable talent for the sake of short term cost-effectiveness or immediate appealing metrics... AIS is going to have a chunk of a problem in a few years without good talent available.
Recently when talking to new people interested in "AI Safety" I’ve been feeling there’s less “alignment” in the sense of caring about x-risk or extinction level issues and working on them as obvious much higher than anything else priority.
This is related to context issue, it’s harder to understand these issues and come to care about them, takes context and time. I’d focus on that a lot re the talent bottleneck: talent that doesn’t get the motivation can’t upskill or work on the right things autonomously, so we need to get them understanding and caring about existential risks.
I think I have the opposite experience. Few years ago, I really had to figure out in advising calls whether someone cared about x-risk, or more mundane risks. Now the usual default is that people are freaked out about large-scale risks.
Unfortunately, most of the orgs themselves don't care about/understand existential risk either. Most of the ones that do have pivoted to advocacy work. There are very few who actually understand the core problems and are actively working on them.
Not my area, but a well written concise post, with great explanation. How right you are I do not know (Not my area again) but I appreciate clear arguments that can easily be engaged with.
Hi Gergő! I believe there is an important misunderstanding when we say AIS talent needs more context.
When hearing this, most candidates believe they need more "technical" or "existential-risk" context (leading them to apply for more BlueDot style courses or workshops). But, is it that so? Or do they need more "field" context?
Are organizations looking for candidates to signal more awareness about AIS urgency? Or rather, they are expecting talent to signal values alignment and knowledge of AIS and adjacent-fields (EA frameworks, network connections, current AIS organizations needs and bottlenecks, etc.)?
In other words, are AIS organizations looking for capable and informed talent about AIS dangers, or are they prioritising talent they can trust and informed about the field particular needs?
These are very different context requirements, and both can be important. But maybe, we are failing to properly inform and guide talent about the latter.
I also believe an important constraint in AIS is the scarcity of organizations to absorbe promising talent and funding available. Maybe that is also the reason why is getting harder to enter the field as an outsider. Should we focus on building more high-impact AIS organizations instead of demanding and increasing higher bar for talent?
I have experienced both sides of the candidate and hiring manager equation, and I know there is not a clear answer to all this. But just telling promising talent to have more "context" without proper guidance and nuances seems not being enough.
Thanks for sharing this! I didn't think of people misunderstanding what "we" mean by context, but if it leads to taking technical courses unnecessarily, that sounds like a problem indeed. What I mean by it is outlined here, which is where I usually send people, including in this post. It gives people quite a lot of pointers on what to do!
An important point, Gergo. I also agree with the comments about unpacking “context”. As someone making this transition, I’d find it useful to know what kind of context a role needs, to what depth, and what can be learned after joining.
Understanding AI risks and the landscape is one layer; deeper knowledge of stakeholders, research agendas or ethical questions will depend on the role and organisation. Clearer expectations would help people direct their effort. It’s understandable that a growing field is still working this out, but losing capable people to confusion and disillusionment is a real risk.
Great post, Gergo! I have noticed the same thing about people not being able to get into a BlueDot course, which used to be the lowest bar to clear into AI safety. Interestingly, despite having context and facilitation experience, I also wasn't accepted as a facilitator, which you would expect is the main bottleneck to getting more course participants (although this could just be that my availability didn't match the demand of participants).
I will also say that, as someone who's full time into AI safety, it's still hard for me to keep up with the context! I know this could be solved by asking any AI agent to get me a summary of the latest news, but sometimes I don't have time even for that (:
Quick typo note on the last paragraph: "...which makes them hard to fill even when fill even when funding..." should probably read "...which makes them hard to fill
even when filleven when funding..."Thanks for the thoughtful comment and bringing the typo to my attention!
I'm sure I'll get downvoted for this, but it has to be said. This seems like the exact opposite approach we should be taking. The alignment field has catastrophically failed by every measurement, and hasn't produced anything viable since RLHF. Whatever context these people have, it seems to be the wrong kind. “Seniors” in the field should not be looked up to as mentors to emulate, but as cautionary tales for new entries of how your thinking, your decisions, and your actions can all go horribly wrong.