Three Interventions, Same Structural Patterns

In my last post, I ran a multi-agent pipeline against GiveWell’s water chlorination cost-effectiveness analysis and got an 84% signal rate — the share of AI-generated critiques worth investigating — compared to GiveWell’s 15-30% baseline. Zero hallucinated citations. Quantified impact tied to actual spreadsheet parameters.

That was one intervention. The obvious question was whether the results would hold up or whether water chlorination was a lucky pick.

I’ve now run the same pipeline against all three interventions GiveWell published AI red teaming output for: water chlorination, insecticide-treated nets (ITNs), and seasonal malaria chemoprevention (SMC). The signal rates held. But the more interesting result is what the pipeline found across all three.

The Numbers

Intervention	Critiques Generated	Survived Full Pipeline	Signal Rate
Water chlorination	31	26	84%
Insecticide-treated nets	30	30	100%
Seasonal malaria chemoprevention	34	28	82%

Every run produced novel critiques absent from GiveWell’s published AI output. Zero hallucinated citations across all three. Each surviving critique includes parameter mappings to specific spreadsheet cells and computed sensitivity ranges.

ITN’s 100% survival rate deserves a caveat — I’ll come back to that.

What Made It Interesting

Individual findings within each intervention are useful but expected. A pipeline with verification and quantification should produce better critiques than a single pass; that’s the whole thesis. What I didn’t expect was how much the findings would rhyme across interventions.

Three patterns showed up independently in all three runs. The pipeline didn’t go looking for them — they emerged from separate investigation threads, generated by separate Investigator agents, verified against separate evidence. The Synthesizer in each run flagged them in its meta-observations, and when I compared the three reports side by side, the convergence was obvious.

Static Parameters for Dynamic Phenomena

All three CEAs model key effectiveness parameters as constants. In every case, the evidence says they change over time.

Water chlorination: adherence decays beyond the 1-2 year RCT durations that generated the base estimates, but the model applies a fixed adherence rate across a 5+ year program horizon. The Verifier found studies showing chlorination adherence drops sharply after even 8-10 week surveillance periods.

ITNs: insecticide efficacy and net physical integrity both degrade, but the CEA applies a single mortality reduction parameter rather than a time-varying effectiveness curve. The pipeline found evidence that the nets used in the original efficacy trials are no longer representative of what’s distributed today.

SMC: drug resistance mutations accumulate across treatment cycles. Post-SMC surveillance in Burkina Faso showed resistance markers rising from 18.6% to 58.3% over a decade — but the CEA contains no resistance adjustment factor, despite GiveWell modeling efficacy decay for other interventions like bed nets.

In each case, time-invariant parameters mean the model overestimates multi-year benefits. The magnitude differs — the SMC resistance finding is potentially the largest, since moving efficacy from 75% to 50% would roughly halve cost-effectiveness — but the structural pattern is the same.

Heterogeneity Hidden by Aggregation

Each model uses aggregate categories that collapse meaningful variation.

Water chlorination: “diarrhea” aggregates pathogen-specific etiologies with very different chlorine susceptibilities. Cryptosporidium, resistant to standard chlorination and responsible for 8-15% of severe childhood diarrhea in implementation areas, gets no separate treatment. “Under-5 mortality” spans an age range where vulnerability changes dramatically — a 6-month-old and a 4-year-old face different risks from different pathogens, but the model treats them identically.

ITNs: “insecticide resistance” conflates two fundamentally different mechanisms. Chemical resistance means mosquitoes contact the net but survive. Behavioral resistance means mosquitoes avoid the net entirely by shifting to outdoor or early-evening biting. GiveWell applies a single resistance adjustment (up to 58.6%) that’s calibrated to chemical resistance — but species composition surveys show outdoor-biting species replacing indoor-biting species in implementation areas, a phenomenon that adjustment doesn’t address.

SMC: “adherence” averages across socioeconomic strata with different access patterns and across monthly cycles with different compliance rates. Evidence suggests the poorest quintile has 6-10 percentage points lower adherence, but the model assumes uniform compliance.

The common thread: aggregation makes models simpler but hides the variance that matters most for cost-effectiveness in specific contexts.

Biological Adaptation (Malaria Interventions)

ITNs and SMC share a domain-specific blind spot: neither accounts for the target organism adapting.

For ITNs, this means mosquito populations shifting toward species that bite outdoors or in the early evening, outside the window when people are under nets. Multiple sites show Anopheles arabiensis (outdoor-biting) replacing An. gambiae (indoor-biting) as the primary vector following net distribution. This is documented, not speculative — but the CEA has no mechanism to capture it.

For SMC, it means the parasites evolving drug resistance under selection pressure from mass administration. The Burkina Faso resistance data is the starkest example, but the pattern appears wherever SMC has been scaled. The CEA’s efficacy estimates come from SMC’s early rollout phase (2012-2018), before resistance markers reached current levels.

Both cases represent models assuming a static biological environment while operating in a dynamic one.

Diagnosing and Fixing a Cost Overrun

The first water chlorination run cost about $30 in API calls — double the $15/intervention target I’d set. The pipeline finished, the results were good, but if every intervention cost $30, three phases would eat the entire $50 budget with nothing left for iteration.

I needed to figure out where the money went.

The pipeline logs every API call with token counts and cost. I pulled the stage-level breakdown and the answer was immediate: the Verifier had consumed about $21 of the $30 total. The Decomposer and Investigators together cost under $1. The Quantifier cost about $7. The Adversarial Pair and Synthesizer cost about $2. The Verifier was 70% of the budget.

Why? The original architecture called for verifying each critique individually — one API call per critique, each with web search enabled. That made sense from a design standpoint: the architecture spec says “the Verifier sees one critique at a time” to prevent batch-confirming based on pattern rather than evidence. But it meant 31 separate API calls, each carrying the full system prompt plus web search context, at roughly $0.67 per critique.

The fix had two parts. First, I batched critiques by investigation thread — sending 2-3 related critiques to the Verifier per API call instead of one. Critiques from the same thread share context, so the Verifier can search once for background on a topic and apply it to multiple hypotheses without re-fetching. Second, I added search count limits and domain focusing to reduce the per-call web search overhead.

The result: ITN verification cost about $5.97 for 30 critiques (roughly $0.20/critique). SMC cost about $8.57 for 34 critiques ($0.25/critique). That’s a 65-70% reduction in per-critique verification cost.

There’s a trade-off. Batching means the Verifier might anchor on its assessment of the first critique when evaluating the second. The architecture spec warned about this. In practice, the verification quality didn’t visibly degrade — the ITN and SMC runs produced comparable evidence detail and appropriate rejection rates at the Verifier stage. But I’d want to test this more rigorously before calling it settled.

The broader lesson: in any pipeline where a stage uses tool calls — web search, code execution, database queries — that stage will dominate your costs, and you won’t know by how much until you’ve run it once. The Phase 1 overrun wasn’t wasted money. It was the cost of learning where the bottleneck was, which made Phases 2 and 3 feasible. If I’d tried to optimize before running, I would have guessed wrong about which stage to optimize.

Final costs:

Run	Cost	Notes
Water chlorination (v1)	~$30	One-at-a-time verification; identified the bottleneck
Water chlorination (v2)	$2.36	Re-ran adversarial + synthesizer only after Quantifier fix
ITNs	$16.00	Batched verification
SMC	$15.96	Batched verification
Total	~$34	Under the $50 budget

What Didn’t Work

Honesty requires flagging what the pipeline does poorly.

The Quantifier’s mapping problem. Most critiques target multi-parameter concepts — “temporal dynamics of adherence decay” or “within-category pathogen heterogeneity” — that can’t be expressed as changing one spreadsheet cell. In practice, only 2-3 critiques per run mapped cleanly to single-parameter perturbations. The rest got proxy calculations that are directionally informative but not rigorous sensitivity analysis. This is the pipeline’s weakest stage.

The adversarial stage is too lenient. ITN’s 100% survival rate and SMC’s 97% survival rate suggest the Challenger agent isn’t filtering aggressively enough. A well-calibrated adversarial stage should reject some fraction of critiques — if everything passes, the bar is too low. Tightening the Challenger’s prompts or adding a stricter elimination threshold on surviving strength would be a priority for a production version.

GiveWell’s spreadsheets resist automation. All three CEA files required per-intervention subclasses with manual parameter mapping due to inconsistent layouts, merged cells, and Google Sheets functions that export as dummy placeholders. This isn’t a pipeline problem exactly, but it constrains the Quantifier to operating on parameter snapshots rather than live formula chains, limiting its ability to capture parameter interactions.

What This Adds Up To

The Phase 1 result could have been a fluke — one intervention, one good run, maybe I got lucky. Three interventions with signal rates of 82-100%, consistent cross-intervention patterns, and zero hallucinations makes a stronger case.

The thesis from the first post holds: GiveWell’s AI red teaming limitations are architectural, not model-level. Decomposition, scoped context, verification, and quantification each independently improve output quality, and they compound. The pipeline uses only public materials and commodity models — Claude Sonnet for most stages, Opus for decomposition and synthesis. The improvement comes from methodology, not privileged access or frontier capabilities.

The cross-intervention patterns are the new contribution. A single-pass approach can find issues within one analysis, but it can’t see that the same structural tendency — treating dynamic phenomena as static, collapsing heterogeneity into averages, ignoring biological adaptation — appears across GiveWell’s entire modeling framework. That kind of pattern recognition requires running the pipeline multiple times and comparing results, which is exactly what a systematic methodology makes possible.

What’s Next

GiveWell’s AI red teaming covered six interventions. I ran three — the three for which they published detailed AI output, which the pipeline uses for novelty filtering and baseline comparison. The other three are community-based management of acute malnutrition (CMAM), syphilis screening and treatment, and malaria vaccines.

The pipeline doesn’t strictly need GiveWell’s AI output to run. It feeds into two places: helping the Decomposer avoid re-raising known issues, and giving the Synthesizer a baseline for the comparison table. Without it, the pipeline still produces the same critiques — you just lose the filter against what’s already been found and the side-by-side scoring.

I’m reaching out to GiveWell to ask whether they’d like me to run those three. If they have AI output they can share, the pipeline can use it to focus on novel findings. If they’d rather see what the pipeline produces blind — without any prior filtering — that’s a valid test too and might be more informative about what the methodology catches independently. Either way works.

Everything from the first three runs is open source: the pipeline, the prompts, the results, and the manual workflow that requires no code.

github.com/tsondo/givewell_redteam

Same channels if you want to discuss: [email protected], BlueSky @tsondo.com, or via the about page.