This is a crosspost for this post I created with Claude Opus and Sonnet 5.5.
Bottom line: cutting small risks can raise large risks in the same category, but mostly through physical accumulation (wildfire), behavioural compensation (levees) or masking (proxy metrics) — rarely through lost salience, the channel the AI-slowdown argument relies on. Fields that reduce large risks almost never respond to the salience worry by tolerating small harms; they respond by capturing the signal another way (incident reporting, near-miss analysis, communication). The one major exception, fire management, deliberately allows small fires because small fires physically remove fuel, not because they make fire risk salient.
1. The claim and when it holds
Jordan Arel's 9 reasons against a near-term AI slow-down argues (reason 1): "A near-term slowdown could prevent the warning shots that would make the public take AI risk more seriously." In a reply in the comments: "the risks seem like they will get much worse over time, so an earlier warning shot could be high value if it reduces risk skepticism for the bigger risks."
Reducing small risks affects the large risk through two channels. A simple linear model shows how they combine:
The coefficients b, c and e are non-negative numbers:
- b (direct channel): how much large risk rises per unit of small risk at fixed safety effort. It is positive when small and large risks share causes. Cutting small risks then also cuts large ones.
- c: how much large risk falls per unit of safety effort.
- e: how much safety effort rises per unit of small risk, i.e. how strongly small events prompt more effort.
- c × e (salience channel): the large risk avoided through the safety effort that small events prompt.
Cutting small risks raises the large risk only if c × e is greater than b, i.e. only if the salience channel outweighs the direct channel. That requires conditions 1–3 below: a large e (condition 1), a large c (condition 2) and a small b (condition 3). Condition 4 does not affect the sign: it decides only whether tolerating small risks is worth it even when the salience channel dominates (and cheap substitutes, once adopted, reduce e).
- Small events substantially raise safety effort (salience is large and does not decay).
- That extra effort is well directed at the large risk (not at the visible small one).
- Small and large risks share few causes. Cutting small ones then does little directly.
- The salience cannot be bought more cheaply without the harm (reporting, demonstrations, simulations).
Four other mechanisms can also make cutting small risk raise large risk, and they need none of conditions 1–3:
- Accumulation: small events discharge a stock that otherwise feeds big events (fuel, leverage).
- Compensation: feeling safer, people add exposure or drop precautions (levees, Peltzman effect).
- Masking: small harms are suppressed at the symptom level while the cause persists, degrading the signal (BP, AI chain-of-thought training).
- Overcorrection: avoiding the small event pushes behaviour toward a riskier failure mode.
2. Other domains
The effect is real in several domains, but the pure-salience version has the weakest evidence; the strongest cases run through accumulation, compensation or masking.
| Domain | Small risk | Large risk | Mechanism | Does cutting small raise large? | Evidence (verbatim) |
|---|---|---|---|---|---|
| Wildfire | Low/moderate-severity fires | Extreme high-severity fires | Accumulation | Yes, strongly | Kreider et al. 2024, Nature Communications: "fire suppression ensures that wildfires will burn under extreme conditions at high severity"; burned area increases "three to five times faster over time" |
| Aircraft landing | Firm/hard landings | Runway excursions | Overcorrection | Yes, if cut by penalising firm landings; no if cut by better approaches | Boeing, ICAO COSCAP-SEA 2024: punishing firm landings makes pilots "focus on achieving a landing that feels soft to the exclusion of other important parameter"; "long flares, missed touchdown zones, etc increase runway excursion risk" |
| Floods | Frequent floods | Losses from rare floods | Compensation | Yes, for losses (exposure grows behind levees) | Di Baldassarre et al. 2018, HESS: protection "can attract more settlements and high-value assets" and "generate a sense of complacency" |
| Finance | Small downturns | Systemic crises | Accumulation + compensation | Plausible, contested | Minsky, summarised by Economics Help: "periods of calm encourage behaviour that eventually leads to crisis" |
| Industrial safety | Personal injuries (slips, falls) | Process explosions | Masking | Cutting small did not cut large; attention diverted | Baker Panel 2007, via NBC: "BP mistakenly interpreted improving personal injury rates as an indication of acceptable process safety performance" (Texas City, 15 dead) |
| AI training | Visible misbehaviour in chain of thought | Hidden misbehaviour | Masking | Yes (signal lost) | OpenAI 2025: with CoT pressure the agent "still cheats, albeit at a lower rate than the baseline, and almost all of its cheating is undetectable by the monitor" |
| Vaccination | Visible measles, polio cases | Outbreaks after coverage falls | Salience | Yes, weakly (the closest real salience case) | Columbia Public Health 2015, quoting Offit: "When vaccines work, nothing happens. To some extent vaccines have become victims of their own successes" |
| Nuclear power | Valve failures, minor incidents | Core damage | Salience / learning | No: incidents helped only once acted on | Blandford & May, AAAS: the Three Mile Island (TMI) valve failure "had occurred in eleven other instances, but this operational experience ... had not been shared"; TMI then led to the Institute of Nuclear Power Operations (INPO) |
| Pandemics | Regional outbreaks (SARS, Ebola) | Global pandemic | Salience | Salience decays fast | Think Global Health: pandemic preparedness got under one cent per dollar of health aid in 1990–2021 despite repeated outbreaks |
| Near misses (general) | Harmless close calls | Disasters | Salience, often reversed | Often no: near misses breed complacency | Tinsley, Dillon & Cronin, via Schneier: near misses read as "disasters that did not occur" make people "illegitimately underestimate the danger" |
Risk compensation, the mechanism behind the levee and Peltzman cases, is a general caveat for safety measures. Hedlund 2000 states the theory that people "use up" the additional safety "though [sic] more risky actions". The paper "surveys risk compensation by reviewing its history, discussing its theoretical foundations, outlining evidence for and against its claims, and providing the author's own views"; its abstract does not say how large the effect is. No claim about this is made here.
3. Do large-risk reducers worry about this?
Only where small events physically or behaviourally reduce the large risk. Where the link is salience alone, practitioners keep cutting small risks and protect the signal instead.
| Field | Worries that cutting small risk raises large risk? | What they do about it |
|---|---|---|
| Wildfire management | Yes, central to the field ("suppression bias", "fire paradox") | Deliberately allow or set small fires (prescribed and managed fire) |
| Flood management | Yes ("levee effect", "safe development paradox") | Land-use limits behind levees, risk communication; still build protection |
| Macro-finance | Partly (Minsky, moral hazard of bailouts) | Macroprudential rules, stress tests; still dampen recessions |
| Process safety, aviation, nuclear | Yes, but about masking and under-reporting, not about lost salience | Separate leading indicators for catastrophic risk; non-punitive reporting (flight operations quality assurance, near-miss systems, INPO) |
| Vaccination | Yes ("victims of their own successes") | Communication, mandates; nobody proposes tolerating measles to restore salience |
| AI safety | Yes, in two forms: warning shots (this post) and masking (do not train against observed misbehaviour) | Debated; the masking concern has the stronger evidence (OpenAI 2025) |
The revealed consensus: the salience value of small harms is real but almost always cheaper to obtain without the harms. No field identified in this analysis reduces large risk by tolerating small harms for their salience alone. Fire management, the closest analogue, relies on accumulation, which has no clear AI counterpart. This is from the sources linked above and general knowledge of these fields, not a systematic survey.
4. Implications for AI
For the warning-shot argument:
- Real incidents have produced lasting safety institutions: Three Mile Island led to INPO the same year (Blandford & May).
- AI risk scepticism is high, and concrete harms persuade publics and legislators more than arguments.
- If AI risks grow steeply with capability, an early, survivable incident is worth more than a late one (Arel).
- Accumulation has a possible AI analogue: a slowdown may build a compute or algorithmic overhang. Capabilities would then jump faster later (speculative, contested).
Against:
- Shared causes: small and large AI harms both stem from weak alignment, weak security and racing; reducing those reduces both, making the direct channel large.
- Salience decays: the panic-and-neglect pattern in pandemics shows attention fading within a few years.
- Near misses often reassure: a harmless AI incident can read as "disaster that did not occur" (Tinsley, Dillon & Cronin).
- Misdirected response: incidents provoke fixes to the visible failure, as in BP's personal-safety focus, not to the large risk.
- Salience has cheaper substitutes: dangerous-capability evals, model-organism demonstrations, mandatory incident reporting.
- A slowdown is not mainly a small-risk reduction: it also buys alignment and governance time, which cuts large risk directly.
Conclusion: the argument is valid in form, but its three sign conditions (section 1) look unlikely to hold jointly for AI. A rough judgement (speculative) is that cutting small AI risks raises large AI risk only in the masking case. That calls for refusing to hide small failures, not refusing to prevent them. Practical upshots:
- Fund incident databases and mandatory disclosure to make small failures that do occur maximally salient.
- Stage "controlled warning shots": public, harmless demonstrations of dangerous capabilities or deception.
- Avoid training against monitored misbehaviour; track catastrophic-risk indicators separately from everyday harms.
- Judge the overhang concern on its own evidence; it is an accumulation argument, not a salience one.
I actually think this is a consideration that is very underrated by people trying to reduce long-term risks. Some related references/thoughts here and there. Some people at CLR a few years ago were calling this the "Smokey Bear effect", in reference to campaigns against wildfires that ended up preventing benign wildfires and causing bigger ones (for the same reasons you discuss above, I think?).
I haven't taken the time to engage with your arguments for why pausing AI might overall reduce AI x-risks regardless, but I'm initially skeptical, fwiw. It seems more complicated than what you suggest. (I might go back to this and engage some time in the future.)