Sudhanshu Kasewa
Posts 2
Comments37
IANAGM so I can't speak to why they are funding or not funding the specific bets they are/are not funding.
But, speaking abstractly, grantmaking in AI safety is conceptually complicated: Do you take a wide range of bets on unproven theories of change and new grantees, or do you narrowly fund only bets which you have high confidence on? How do you balance between the two?
My intuition is that AI safety puts 1-2 cycles of funding into first-time bets, and quickly moves on to the next set of first-time bets while only continuing to fund orgs and people who used their early money well (and had something to show for it, even negative results). One could argue that funders should be biased towards funding orgs/people for 3 or even 5 cycles before pulling the plug, but this is ultimately a hyperparameter in a quantitative model. It's not obvious to me that the current settings of such hyperparameters are the wrong ones.
With more funding and increased diversity of funders, I absolutely expect more funding philosophies and models in the ecosystem, increasing the kinds of bets being taken.
On "How", my colleague Matt @Matt Beard has some good ideas here: https://80000hours.substack.com/p/how-to-get-into-ai-safety-in-3-months
Yes, it can take time, but one can say more:
- Not everyone has taken a long time, and among my observations, this is usually because they have demonstrated exceptional bias for action. E.g. Julian Moncarz , Neav Topaz (maybe it's no coincidence that they are both at Kairos!)
- I think on priors one should expect pivots to take time, in any domain. The kinds of things one has to do could be different, e.g. getting an MBA to break into management, or a PhD to get into particle physics. Actually AI safety is much, much less gate-kept since there is no hard experience, degree, location requirements
- AI safety looks as if you need to take a lot of time to pivot compared to e.g. pivoting to some mature industry, but this under-counts the actual pivot time for the latter: Mature industries have huge companies with a lot of slack in them which can take on promising but relatively unproductive pivoters and put them through years of on-the-job upskilling. (but obviously having the security of a full-time job is much preferred to fellowships or career development grants).
AI safety needs people everywhere but quickly stated, current talent bottlenecks to me look like:
-- Founders
-- Grantmakers
-- (technical) Research leads
-- Policy entrepreneurs and implementors (which includes a lot of technical work)
-- bets in international coordination and/or cooperation
-- All manner of supporting talent -- program leads, ops proper, public outreach, content creators, comms
Most sought-after qualities for talent are:
-- context, mission alignment, domain understanding, sophisticated views on AI strategy and threat modelling etc.
-- "good judgement", "sound epistemics", "reasoning transparency" and other similar ideas/meta-skills from the EA/rationalist cannon
-- a willingness to get shit done/bias for action (rather than be in learning mode, or people who need a lot of management and oversight, or folks with too many preferences/constraints)
-- low ego, similar to above
-- ambitious folks, since they would be really trying to be their own managers, take on bigger projects, grow themselves and their teams etc.
Finally, even having these, it's not enough to just claim to have these; job-seekers mainly trip up in being able to demonstrate and be legible about having them.
Weekly Prompts
Recently, an advisee told me that they've been procrastinating on replying to my email. It sits at the top of their stack each week. When they try to reply, instead they act on the prompts within, and so no longer need to correspond with me for the time-being.
They run this in a loop, and keep moving forward.
My email:
(1) Can you write out, say, 5 questions that you have uncertainty about? What would answers to these questions mean for your decision? (It’s important to pick questions/uncertainties that are actually decision-relevant, such that a different answer to those questions would change the decision you make.)
(2) Can you come up with 2-4 actions that can reduce that uncertainty/help get you more clarity? Are any of those actions particularly cheap? (Maybe reading a particular thing, talking to someone in a particular role or organisation?).
Consider setting up such prompts for your own weekly check-ins. Let me know some of your most effective prompts in the comments!
Got sent a set of questions from ARBOx to handle async; thought I'd post my answers publicly:
- Can you explain more about mundane utility? How do you find these opportunities?
- Lots of projects need people and help! E.g. Can you contribute to EleutherAI, or close issues in Neuronpedia? Some more ideas:
- Contribute to the projects within SK’s github follows and stars
- Make some contributions within Big list of lists of AI safety project ideas 2025
- Reach out to projects that you think are doing cool work and ask if you can help!
- BlueDot ideas for SWEs
- From a non-coding perspective, you could e.g.
- Facilitate BlueDot courses
- Give people feedback on their research proposals, drafts, etc.
- Be accountability partners
- Offer to talk to people and share what you know with those who know less than you
- Check out these pieces from my colleagues:
- How to have an impact when the job market is not cooperating by Laura G Salmeron
- Your Goal Isn’t Really to Get a Job by Matt Beard
- Lots of projects need people and help! E.g. Can you contribute to EleutherAI, or close issues in Neuronpedia? Some more ideas:
- What is your theory of change?
- As an 80k advisor, my ToC is “Try and help someone to do something more impactful than if they had not spoken to me.”
- Mainly, this is helping get people more familiar with/excited about/doing things related to AI safety. It’s also about helping them with resources and sometimes warm introductions to people who can help them even more.
- As an 80k advisor, my ToC is “Try and help someone to do something more impactful than if they had not spoken to me.”
- Are there any particular pipelines / recommended programs for control research?
- Just the things you probably already know about – MATS, Astra are likely your best bets, but look through these papers to see if there are any low hanging fruit as future work
- What are the most neglected areas of work in the AIS space?
- Hard question, with many opinions! I’m particularly concerned that “making illegible problems legible” is neglected. See Wei Dai’s writing about this
- More groundedly, I’m concerned we’re not doing enough work on Gradual Disempowerment and more broadly questions of {how to have a flourishing future/what is a flourishing future} even if we avoid catastrophic risks
- In general, AI safety work needs to contend with a collection of subproblems. See davidad’s opinion – A list of core AI safety problems
- There are many other such opinions, and it’s good to scan through them to work out how they’re all connected, so that you can see the forest for the trees; and also to work out which problems you’re drawn to/compelled by, and seek out what’s neglected within those 🙂
- Some questions about ops-roles:
- What metrics should I use to evaluate my performance in ops/fieldbuilding roles? I find ops to be really scattered and messy, and so it's hard to point to consistent metrics.
- Hard to talk about this in concrete terms, because ops is so varied; every task can have its own set of metrics. Instead, think through this strategically:
- Be clear on the theory(ies) of change, and your roles/activities/tasks in it(them). Once you can articulate those things, the metrics worth measuring become a lot clearer
- Sometimes we’re not tracking impact because impact evaluation is notoriously difficult. Look for proxies. Red-team them with people you admire
- Fieldbuilding metrics can be easier to generate, but I don’t claim to be an expert here – ask folks at BlueDot, or the fellowships for better input.
- How many people completed the readings?
- How many people did I get to sign up for the bluedot course?
- How many of those finished the bluedot course?
- How many people did I get into an Apart Hackathon?
- Did any of my people win?
- And so on…
- Hard to talk about this in concrete terms, because ops is so varied; every task can have its own set of metrics. Instead, think through this strategically:
- Likewise, I have a hard time discerning what “ops” really means. What are the best tangible “ops” skills I should go out of my way to skill up on if I want to work in the field building/programmes space? Are there “hard” ops skills I should become really good at (like, familiarity with certain software programmes, etc)
- Ops is usually a “get stuff done” bucket of work. Yes, it can help to have functional experience in an ops domain like “Finance” or “IT/office tech infra/website” (and especially “Legal”), but a LOT of ops can be learned on the job/on your own; AI safety is stacked full of folks who didn’t let “I don’t know anything about ops” stop them from figuring it out and getting it done
- What metrics should I use to evaluate my performance in ops/fieldbuilding roles? I find ops to be really scattered and messy, and so it's hard to point to consistent metrics.
- Under what circumstances should a “technical person” consider switching their career to fieldbuilding?
- First things first:
- Fieldbuilding is not a consolation prize. Do fieldbuilding if you’re really passionate about helping AI go well, and fieldbuilding is your comparative advantage.
- And doubling down on that:
- It really really really helps if fieldbuilders are very competent. A fieldbuilder who doesn’t know their shit about AI risk and AI safety can propagate bad ideas among the people they’re inducting into the field.
- This can have incredibly high costs
- Pollutes the commons
- Wastes time downstream where all this would need to be corrected
- Bounces people who might be able to quickly get up to speed, because their initial contact with these fieldbuilders is of poor quality, poor argumentation, poor epistemics
- This can have incredibly high costs
- Conversely a great fieldbuilder is one who knows how to tend their flock, what they need to prosper and grow to become competent at thinking about AI safety properly, and being able to do AI safety things
- It really really really helps if fieldbuilders are very competent. A fieldbuilder who doesn’t know their shit about AI risk and AI safety can propagate bad ideas among the people they’re inducting into the field.
- First things first:
- How would you recommend going about doing independent project work for upskilling in-place of doing something like SPAR or MATS?
- Why not both? In general, I want people to ask themselves this question when making decisions. You can do a lot more than you give yourself credit for.
- At the current margins SPAR, MATS etc. are probably better than independent work
- Some of these fellowships have pretty high signal to employers (based on evidence that has been generated over time)
- There is a lot that these fellowships offer that are sometimes hard to get without them
- Research support, mentorship, community engagement, well-scoped projects with deliverables and accountability
- Also softer things like physical space , some money
- Research support, mentorship, community engagement, well-scoped projects with deliverables and accountability
- But if you’re great at doing stuff independently, go for it! Neel Nanda didn’t need a fellowship.
- A key idea is to keep your eye on the ball – be productive!
- The point is generate outputs
- That make you learn
- That show that you have learned
- That are related to AI safety
- That get feedback
- That show that you update based on (relevant/good/high-quality) feedback
- The point is generate outputs
- A key idea is to keep your eye on the ball – be productive!
lfg!
Following on from this post:
A few more things I often say that obliquely relate to networking:
- Build in public and seek feedback. People like engaging with stuff online. Do stuff in public -- writing, github projects, YouTube shorts, anything -- and ask people for feedback. Repeatedly doing interesting things increases the quality of feedback and engagement in a positive cycle, and opens doors for deeper collaboration. Consider being pseudonymous if it is stressful to do it with own name.
- Offer your services, volunteer, collaborate. Stolen from Laura's piece on How to have an impact when the job market is not cooperating
- Host events to attract the kind of people you want to engage with. IIUC this is how Mox SF started.
- Consider doing things in-person. Lots of relationships are forged in post-event social settings.
Scrappy note on the AI safety landscape. Very incomplete, but probably a good way to get oriented to (a) some of the orgs in the space, and (b) how the space is carved up more generally.
(A) Technical
(i) A lot of the safety work happens in the scaling-based AGI companies (OpenAI, GDM, Anthropic, and possibly Meta, xAI, Mistral, and some Chinese players). Some of it is directly useful, some of it is indirectly useful (e.g. negative results, datasets, open-source models, position pieces etc.), and some is not useful and/or a distraction. It's worth developing good assessment mechanisms/instincts about these.
(ii) A lot of safety work happens in collaboration with the AGI companies, but by individuals/organisations with some amount of independence and/or different incentives. Some examples: METR, Redwood, UK AISI, Epoch, Apollo. It's worth understanding what they're doing with AGI cos and what their theories of change are.
(iii) Orgs that don't seem to work directly with AGI cos but are deeply technically engaging with frontier models and their relationship to catastrophic risk: places like Palisade, FAR AI, CAIS. These orgs maintain even more independence, and are able to do/say things which maybe the previous tier might not be able to. A recent cool thing was CAIS finding that models don't do well on remote work tasks -- only 2.5% of tasks -- in contrast to OpenAI's findings in GDPval suggests models have an almost 50% win-rate against industry professionals on a suite of "economically valuable, real-world tasks" tasks.
(iv) Orgs that are pursuing other* technical AI safety bets, different from the AGI cos: FAR AI, ARC, Timaeus, Simplex AI, AE Studio, LawZero, many independents, some academics at e.g. CHAI/Berkeley, MIT, Stanford, MILA, Vector Institute, Oxford, Cambridge, UCL and elsewhere. It's worth understanding why they want to make these bets, including whether it's their comparative advantage, an alignment with their incentives/grants, or whether they're seeing things that others haven't been able to see yet. (*Some of the above might be pursuing similar bets to AGI cos but with fewer resources or with increased independence etc.)
(v) Orgs pursuing non-software technical bets: e.g. FlexHEG, TamperSec
(B) Non-technical or less technical, but still aimed (or could be aimed) at directly** working the problem
(i) Orgs that do more policy-focussed/outreach/advocacy/other-non-technical things: e.g. MIRI, CAIS, RAND, CivAI, FLI, Safe AI forum, SaferAI, EU AI office, CLTR, GovAI, LawAI, CSET, CSER
(ii) AGI cos policy and governance teams, e.g. the RSP teams, the government engagement teams, and maybe even some influence and interaction with product teams and legal departments.
** "directly" here means something like "make a strong case to delay the development of AGI giving us more time to technically solve the problem", a first-order effect, rather than something like "fund someone who can make a case to delay...", which is a higher order effect
(C) Field-building/Talent development/Physical infrastructure
(i) Direct talent development: Constellation, Kairos, BlueDot, ARENA, MATS, LASR, Apart Research, Tarbell, etc. These orgs aim to increase the number of people going into above categories or speed them up. They don't usually (aim to) work directly on the problem, but sometimes incidentally do (e.g. via high quality outputs from MATS). There can be a multiplier effect for working in such orgs.
(ii) Infra: Constellation, FAR AI, Mox, LISA
(iii) Incubators: e.g. Seldon Labs, Constellation, Catalyze, EF, Fifty-Fifty
(D) Moving money
(i) Non-profit/philanthropic donors: e.g. OpenPhil, SFF, EA Funds, LongView, Schmidt Futures
(ii) VCs: e.g. Halcyon, Fifty-Fifty
For added coverage,
(E) Others
(i) Multipolar scenarios: CLR, ACS Prague, FOCAL (CMU), CAIF
(ii) Digital consciousness type-things: CLR, Eleos, NYU Center for Mind, Ethics, and Policy
(iii) Post-AGI futures: Forethought, MIT FutureTech
(F) For-profits trying to translate AI safety work into some kind of business model to validate research and possibly be well situated should more regulation mandate evals, audit, certifications etc.: e.g. Goodfire, Lakera, GraySwan, possibly dozens more startups + big professional services firms would be itching to get in on this when the regulations happen.
It is very worth investigating whether to work on any of these: The field is wide open and there are many approaches to pursue. "Defence in depth" (1, 2, 3) implies that there is work to be done across a lot of different attack surfaces, and so it's maybe not so central to identify a singular best thing to work on; it's enough to find something that has a plausible theory of change, that seems to be neglected and/or is patching some hole in a huge array of defences -- we need lots of people/orgs/resources to help with finding and patching the countless holes!