Hide table of contents

[Subtitle.] I interviewed dozens of biosecurity experts. The fear that AI will enable lone wolves to build pandemics is a case study in how domain expertise gets sidelined.

This is a crosspost for Why AI-assisted bioweapons won't kill us all by Abi Olvera, which was published on Positive Sum on 20 July 2026. Here is a response from Olivia Scharfman, and here is Abi's reply to it.

Some of my friends think we are going to die because AI will make building a bioweapon easier.

The fear is often that AI will walk an ordinary person through unleashing a pandemic. Or that AI will facilitate the creation of an engineered pandemic that could kill millions or billions on Earth.

I spent ten years as a US diplomat. My job was often to check claims before Washington acted. I wanted to double check this too.

I interviewed dozens of biosecurity experts who had decades at the lab bench.

Yes, AI makes virology work a little easier. But it is still extremely difficult. Media headlines often say “AI can make this step easier” without the broader context that a bioweapon involves a lot of hard steps that are not easier.

The public doesn’t hear this context because of our information ecosystem.

Let me first explain why I’m no longer worried [about lone wolves and non-state actors]:

Why bioweapons are rare

Since the 1980s, there has been exactly one fatal bioterror attack.[1]

They are rare because 1. they are difficult, and 2. they are bad weapons.

People who want to cause harm still do cost-benefit thinking. For almost any goal they have, a bomb, a gun, a chemical, or a cyberattack is cheaper, faster, and more controllable.

Practitioners flagged several more barriers:

  1. You have to be comfortable killing your own people. Pathogens kill indiscriminately. Few groups are this malicious and extreme. Groups that are tend toward paranoia and rigid hierarchies that make them less effective.
  2. You need significant resources. Bioweapons capable of harming many people need equipment, chemicals, and professionals trained in different fields.
  3. You need an institution for access to specialized equipment and materials. Many items are too niche for black markets. Essential equipment like biosafety cabinets or fume hoods requires professional installation and ongoing manufacturer support. Some materials are restricted; for example, viruses need host cells, cell lines are sold only to institutions, and plasmids require institutional credentials. In the United States, only tens of thousands of people have sustained access to such facilities and supplies.
  4. Your team must be both highly competent and completely secret. A contamination incident could destroy the pathogen or infect the team.
  5. You need to convince and retain experts from different domains e.g. biology, chemistry, engineering, aerosolization, or fieldcraft to carry out attacks. Recruiting without detection is hard too.

For a detailed version of this list, see my full piece.

Even well-resourced groups with scientific talent still fail. Aum Shinrikyo, a Japanese cult with ~$1 billion, failed in all ten of their attempts to release biological agents.

Why AI models don’t change this

AI can supply biology and chemistry knowledge, but not hands-on skill.

Reading every surgery textbook doesn’t make you a good surgeon. The biology laboratory works in a similar apprenticeship model. You learn by failing repeatedly under a mentor who has also failed repeatedly.

To get a sense of the physical nature of biology work, I explored one aspect: moving liquids.

Biologists often use pipettes, very precise eye-dropper-like tools. Students will often try fifty times to get it right. Practitioners develop a feel for specific materials over the years.

woman holding brown glass
Photo by CDC on Unsplash

This skill appears in almost every bioweapon protocol. It is hard because:

Materials are often fragile. Some DNA strands can shear from hand tremors. Some proteins are so temperature sensitive that the pressure of moving the liquid generates enough heat to destroy them.

Concentration has to be exact. If a sample is too dilute, DNA strands may never find each other. Instructions might say “check with a spectrophotometer,” but a practiced scientist recognizes when a bubble or residual chemical is corrupting the reading.

Contamination in biology is unforgiving. A one-in-a-million contaminant in chemistry is usually negligible. In biology, this can matter because organisms multiply.

Creating a bioweapon involves many more complex steps. Converting viral DNA into a live virus is hard even for researchers. Stabilizing a pathogen into an effective aerosol often requires specialists from different fields. AI can describe the process but doesn’t give you the skills to execute it. (I walk through the full chain of steps here.)

Even success probably wouldn’t be catastrophic

A motivated team of scientists with institutional access could keep trying and eventually succeed at making a pathogen.

But success would not necessarily produce a new pandemic. Any pathogen that a team could plausibly assemble is one that already exists in nature. A lab-assembled version of COVID-19 would have about the same effect as an infected person coughing in a crowded room.

The more serious worry is that someone could revive a pathogen no longer in circulation, like the 1918 influenza or engineer something entirely new.

The 1918 flu infected roughly a third of the world’s population and killed an estimated 50 million people. Releasing it today would be very bad. It would not be 1918-bad. Most people’s immune systems recognize flu descendants. We also now have antivirals and antibiotics to treat the secondary bacterial pneumonia that killed most victims. Today’s vaccines offer cross-protection.

Engineering a virus that is both extremely lethal and highly contagious, worse than anything nature has produced, would be bad. But many biologists doubt this is possible even with perfect technology. People have very complex immune systems that often react differently. Viruses mutate as they spread. But more importantly: biology has built-in tradeoffs: the more lethal a virus is, the less transmissible it is (because it kills its human host).

When it comes to designing such a virus, any edits to make a virus deadlier could make the virus unstable or incompatible with human life. In physics, you can tell when an experiment is getting closer to a goal. Biology doesn’t work that way, meaning you can spend a long time on an endpoint that is not biologically possible.

Nature makes new viruses constantly, but by brute force. It spins out countless mutations and lets billions of them die every day.

Why this isn’t conventional wisdom

When I published this research, leaders in biosecurity and national security organizations thanked me privately. “My organization found the same conclusion, hence why we don’t work on that.”

Null results aren’t often published. Looking at the media sphere specifically: “bioweapons are hard” is not news. It isn’t a headline; it’s always been true.

Advocacy organizations have similar constraints. If you are pushing for AI oversight, a finding that undercuts urgency on one specific threat does not help.

Biologists, the people best positioned to explain this, mostly don’t write for general audiences. Academic incentives don’t reward op-eds, especially about things that are obvious within the field.

Our information systems get less accurate as issues become politicized. Alarm, not context, ends up on the news partly because of who is incentivized to speak up.

I’ve worked through the evidence across three Golden Gate Institute for AI pieces, if you want to go deeper:

Update: To be clear, biosecurity is still critical! My point above is narrow, aimed at a claim common in EA and x-risk circles. Even EA-affiliated organizations like CLTR and GovAI have published a similar point: an outsized focus on lone-wolf bioterrorism and engineered pandemics can crowd out other threats, like smaller bioweapons, non-transmissible pathogens, chemical weapons, and explosives. Broad biological resilience helps across many of these scenarios, especially naturally occurring pandemics, which carry a 2-3% chance each year.

Huge thanks to Golden Gate Institute for AI for supporting this research!

I am giving a version of this talk at the Mind and Machine Alignment Summit at Ohio State University tomorrow.

AI and cyber risk is my next AI deep dive. I’d love your favorite resources in the comments! Reach out directly if you’d like to talk.

  1. ^

36

1
6
2

Reactions

1
6
2

More posts like this

Comments11
Sorted by Click to highlight new comments since:

As a biologist, I want to zone in on one particular part of the post here because I find it very off base.

Whatever expert interviews were done to get the list of "tacit knowledge" seems deeply flawed, as the difficulty of the listed examples is overstated by orders of magnitude. I get the impression that this list was compiled by talking to ancient PIs that haven't done experiments in decades and have lost much of their sense of proportion. This is a semi-common phenomenon.

Specifically,

  1. Pipetting isn't difficult. You push a button up and down. Students get it in a few tries. Pipetting volumes extremely precisely requires some days of practice but is practically never necessary.
  2. Fragile materials exist but are pretty rare and honestly not mythically difficult to handle. The one case I encountered came down to not using the vortex machine on it and pipetting it slowly. This really isn't relevant all that often and it's not super severe when important.
  3. Most biology works fine or a bit less effectively, if the concentrations are off by 20%. If it's not working you just vary the concentrations in a "grid search" till it works
  4. Contamination is a regular problem in lots of wetlab settings. It definitely requires some practice to keep things sterile consistently. That said, students in university microbiology lab classes routinely pull this off first try with just a Bunsen burner.

It's  true that much (but by no means all) biology work requires weeks or months of finicky optimization before you can get it working reliably, but this has little to do with these (to me bizarre) claims.

Pipetting is hard based on the type of task you need to do. Obviously, pipetting is super easy when the thing itself is basic. The difficult is a function of the braoder task you're doing. Fragile materials, concentrations, contamination: all this is to illustrate what "lab hands" mean to non-bio people people, as well as why some things in textbooks are much harder in real life.

None of this is to say that making a basic bioweapon is impossible for people who have an institution, basic skills, the ability to order controlled things, and the desire. The people who have a list of all of those things, plus the skills for very specific types of virus sets, are a smaller crowd, in general, which people seem to not be understanding. Hence, see my BOTEC of those people in Article 1.

I'd love to know what "bizarre" claims you mean. A good number of the biologists were actively doing wet lab work. It's been publicly commended predominantly biologists and domain experts.

For me the "bizarre" part continues to be the focus on the manual execution of the specific tasks you listed.

To try and double check that this isn't a matter of me having a blind spot or miscommunicating the idea, I presented a colleague (first person I found after seeing your response, no other special selection besides me knowing them) with your list and asked if they thought that these would be key points that would stop a fresh student from completing an experiment assigned to them.

Here, I see that my phrasing was maybe suboptimal as there was some confusion and back and forth. Initially, this responded that this could be a big deal, as you need to be aware of them to handle an experiment. However, when I specified that you were allowed to inform the student of it and provide a protocol, they said the student would do fine in that case and denied any significant hurdle involving "manual" learning. So after this sanity check, I stand by my claim that while people aren't necessarily aware of the relevance of these things, actually physically executing them isn't that hard.

My colleague also independently suggested (with no other context) that Americans might be more prone to perceive these things in terms of manual skill, as according to my colleague, they tend to be left to their own devices more and have less supervision during their PhD years, so these things might present more of a perceived learning curve.

Are you familiar with the term "lab hands"? This is the term used in biology lab circles to call what I'm trying to convey. Do you contest that there's some people who have more success in lab work because of something people just simplify by saying this term?

No I don't contest that. My point of contention is that the specific points above don't fall under this for me.

Comparison: You would expect someone who has been in a kitchen once in their life to fail somehow at making a stew from a recipe, but it would be wrong to say that they couldn't do it because learning to stir a pot is difficult and takes lots of training.

Beyond just accuracy, I think this is relevant here because purely manual skills like "stirring a pot" isn't something more AI supervision/involvement can accelerate, whereas issues like "how do I turn on the stove" etc. are more amenable to being fixed by more supervision.

I do agree with the broad claim that it's unlikely we'll see lone wolf attacks of this nature.

I don't think I'm claiming primarily/solely manual dexterity, so I may have communicated this not super well.

The barrier I'm describing is that protocols are underspecified in ways that can't be fixed by writing more words: "gently" has no operationalization that transfers across materials. The optimal "gentleness" often depends on what the specimen is doing that day too. There's some level of knowledge that is very hard to write ahead of time, also because each specimen is different. This is harder to fix.

This post feels like it is saying that lone actors are currently incapable of creating an X-risk bioweapon. I agree. I think almost everyone in the X-risk scene agrees.

The concern is instead about whether this capability will exist soon. To show that being impossible (or very unlikely), IMO you need to show that there is at least one critical step that is not getting democratized. IE, bio-expertise is getting cheaper because of increasingly powerful open source models. Robots with sufficient dexterity to do bio experiments are getting more capable and cheaper. Where is the bottleneck that will remain unsolved for 2+ decades, and why are we confident it stays unsolved?

To give an analogy, 2 years ago non-experts couldn't solve major open math problems. Today, with access to frontier models, they can. How can we be confident that the bottlenecks against independently creating novel & powerful plagues will not also fall?

Hi Dan. I expect terrorist attacks to be super unlikely to cause human extinction in the absence of very strong evidence suggesting otherwise. For the attacks killing more than 10 people among the 202 k listed on the Global Terrorism Database (GTB), the fraction killing more than 10*N people is around 1 % of the fraction killing more than N people. This suggests a Pareto distribution with a tail index of 2. Around 0.001 % of attacks killed more than 1 k people. Assuming the same Pareto distribution (straight line fitted to the section of the graph below with more than 10 deaths), more than 1 billion deaths would be 10^-12 (= 0.01^(9 - 3)) times as likely as more than 1 k deaths. So the Pareto distribution implies only 10^-17 (= 10^(-5 - 12)) of terrorist attacks cause more than 1 billion deaths. I think the actual risk is way higher. At the same time, I believe a significant risk of a terrorist attack causing human extinction, like 1 % over the next 20 years, would depart a lot from the most natural uninformed ways of extrapolating past data. A pandemic causing human extinction looks much more plausible based on past data (although I still guess the risk of human extinction in the next 20 years is much lower than 1 %).

Oh thanks for adding concrete figures, that is very helpful (I should have done the same)!

  1. I agree that the risk of extinction from a terrorist attack is under 1% over the next 20 years, barring a major distribution shift.
  2. My concern is that AI advancements create a real risk of distribution shift. I think we're already seeing some uplift, but I agree we are still quite far from individual non-experts posing a risk (https://forum.effectivealtruism.org/posts/S6ydgTdTr8sXkfs9x/the-state-of-bio-uplift-research-in-mid-2026).
  3. Due to 2, I think I might go as high as 0.1% risk over the next 20 years (with incredibly wide error bars). That still seems high enough to seriously consider, although I agree other X-risk concerns are likely even higher.

I interviewed dozens of biosecurity experts. The fear that AI will enable lone wolves to build pandemics is a case study in how domain expertise gets sidelined.

(Emphasis mine)
How many AI experts did you interview? How about AIxBio experts?

Roughly 10, ~8 of which I interviewed at one point at the same time so to red team each other. At least 4 were in AIxbio. More than one were x-risk/ea-prioritization-focused.

Curated and popular this week
Relevant opportunities