I Red Teamed GiveWell's AI Red Teaming

I wasn’t looking for this project. I was doing research for an AI Ethics course — my teacher had pointed me to a GiveWell article about unintended consequences of charitable interventions. While I was on their site, I noticed they’d published a report on using AI to critique their own research.

GiveWell, if you’re not familiar, is the organization that tries to figure out which charities do the most good per dollar. They’ve directed over $2 billion to programs like malaria bed nets, water chlorination, and malnutrition treatment. Their work runs on cost-effectiveness analyses — spreadsheets that model how many lives a dollar saves through a given intervention. Getting those models right matters a lot.

Their AI red teaming experiment was a smart idea executed in a straightforward way: give ChatGPT their research, ask it to find problems, have humans review the output. They published the results honestly — about 15-30% of the AI’s critiques were worth investigating, but the rest were noise. Repeated concerns already in their report. Hallucinated citations. Made up specific numbers with no basis. Lost track of what GiveWell’s models actually do versus what the academic literature generally discusses.

Reading that list of failure modes, I recognized every single one. Not because I’m an AI expert, but because these are architecture problems, not model problems. They’d asked one model to do six different cognitive tasks simultaneously — generate hypotheses, find supporting evidence, assess quantitative impact, check its own work for novelty, and produce a structured report. That’s like asking one person to be the researcher, fact-checker, analyst, devil’s advocate, and editor all at once, in a single sitting, with no notes between steps.

The fix is the same as in any organization: divide the labor and build in verification.

The Pipeline

I built a multi-agent pipeline that breaks GiveWell’s single prompt into six specialized stages. Each stage has one job, sees only the information it needs, and passes structured output to the next.

The Decomposer reads GiveWell’s intervention report and cost-effectiveness model, then identifies 5-8 specific areas to investigate. It also produces an exclusion list — things GiveWell already addresses — so downstream agents don’t waste time rediscovering known issues.

Investigators (one per area) generate hypotheses about what might be wrong. The critical design choice: they are explicitly prohibited from citing evidence. They identify concerns worth checking, period. This is the single biggest difference from GiveWell’s approach, because it eliminates the main source of hallucinated citations. When you ask a model to simultaneously generate an idea AND provide supporting evidence in one pass, it will invent evidence that fits. When you tell it “just tell me what to look into, a fact-checker will handle the rest,” the hallucination pressure disappears.

The Verifier takes each hypothesis and searches for real evidence — actual papers, actual data, actual guidelines. It classifies each critique as verified, partially verified, or unverified. Unverified critiques are logged transparently, not mixed in with supported ones.

The Quantifier takes verified critiques and runs them against GiveWell’s actual spreadsheet. Not “this could reduce cost-effectiveness by 15-25%” with no basis — actual sensitivity analysis. Change the parameter, trace through the formulas, report what happens to the bottom line. This is the difference between “sounds concerning” and “changes the answer by 12%.”

An Adversarial Pair — one agent defends GiveWell’s current position, another argues for the critique. Critiques that collapse under pushback get filtered out. Critiques that survive are stronger for having been tested.

The Synthesizer ranks everything that survived: verified evidence, quantified impact, debate outcome, recommended action.

The Results

I ran the pipeline against GiveWell’s water chlorination cost-effectiveness analysis. The numbers:

The Decomposer identified 7 investigation threads. Seven Investigators produced 31 candidate critiques. The Verifier searched for real evidence and passed 30 of 31. The Quantifier ran sensitivity analysis against the actual CEA spreadsheet. The Adversarial Pair stress-tested each surviving critique. The Synthesizer ranked 26 critiques that survived the full pipeline — an 84% signal rate compared to GiveWell’s 15-30%.

Total cost: about $32 in API calls. Total time: 90 minutes, mostly unattended.

Zero hallucinated citations in the final output.

The top finding was something the spreadsheet revealed, not something any model intuited: GiveWell’s cost-effectiveness model has a “plausibility cap” — a ceiling on the mortality effect — and several underlying parameters are so sensitive that removing the cap swings cost-effectiveness by nearly 600%. The Synthesizer recognized this pattern across multiple critiques and identified it as evidence of systematic parameter optimism masked by the cap. No single-pass approach would produce this finding because it requires running the actual numbers and then reasoning about what the numbers reveal.

Why the Spreadsheet Matters

Here’s something I didn’t expect: the Quantifier stage was the one that broke during development, and fixing it changed everything.

The first run produced a report with plausible-sounding critiques and model-generated impact estimates — “reduces mortality reduction by 10-15%,” “overestimates by 15-25%.” These numbers felt right but weren’t computed from anything. The Quantifier had failed to connect to the spreadsheet engine.

After fixing it, the rankings changed. A critique about non-linear baseline mortality relationships got promoted from moderate to the #1 finding — the spreadsheet showed it matters more than the model guessed. Seasonal disease transmission got demoted from critical to moderate — the spreadsheet showed it matters less. Two entirely new findings about the plausibility cap emerged because they could only be discovered by perturbing real parameters and observing extreme sensitivity.

The lesson: qualitative AI critique is useful but unreliable for prioritization. The model’s intuition about which concerns matter most was wrong in both directions. Grounding critiques in actual calculations isn’t decorative — it’s structurally necessary for correct prioritization.

A Concrete Example

GiveWell’s AI raised a valid concern about water chlorination: that seasonal flooding increases water turbidity, which reduces chlorine’s effectiveness. This is a real issue — the chemistry is well-established. But the AI also cited “Kenya and Malawi focus groups from 2024” as supporting evidence. Those focus groups don’t exist. GiveWell confirmed the citation was fabricated.

Our pipeline handled this differently. The Investigator raised the turbidity hypothesis without citing evidence. The Verifier searched and found real sources — Si et al. 2022 on turbidity effects and WHO guidelines on chlorine demand — and classified the critique as “partially verified: the chemistry is established but the link to mortality outcomes hasn’t been directly measured.” No fake focus groups. Clear labeling of what’s known and what isn’t. The Quantifier then estimated a 35-45% effectiveness reduction during high-turbidity periods, grounded in the CEA parameters.

An Unexpected Direction

Most red teaming looks for reasons an intervention might be less effective than claimed. But the pipeline also surfaced questions about whether GiveWell might be underestimating the effect.

The plausibility cap constrains the mortality estimate based on an indirect model of how chlorination should reduce deaths. The direct evidence from trials suggests a larger effect, and in some program contexts, the cap is binding — actively pushing GiveWell’s estimate down. If there are pathways by which clean water reduces mortality beyond preventing diarrhea — through reduced immune burden, better nutrient absorption, fewer secondary infections — then the cap could be too conservative.

This might reflect a deliberate methodological choice toward conservatism, or it might be an inherited default that hasn’t been revisited. Either way, it’s the kind of finding that matters when it affects where hundreds of millions of dollars go.

A Note on Data Architecture

One practical finding: GiveWell’s CEA spreadsheets are designed for human analysts, not programmatic access. Inconsistent layouts across tabs, external data links, Google Sheets-specific functions that export as dummy placeholders, and different column structures across program tabs made automated sensitivity analysis significantly harder than it needed to be.

If GiveWell — or any research organization — wants AI to engage meaningfully with their models, structuring data for machine readability is a low-cost investment with high returns. A consistent parameter layout, named ranges, and a single summary tab with all key inputs would make both human and AI red teaming more effective.

This Isn’t a Critique of GiveWell

I want to be clear: GiveWell ran a thoughtful pilot and published their results with unusual honesty. They openly reported hallucination rates, noted which critiques were already known, and acknowledged the limitations of their approach. Most organizations wouldn’t publish their AI’s failures alongside its successes.

The point is that having identified AI red teaming as worth doing, the next step is doing it well. The failure modes they observed are solvable with current tools. The 15% signal rate isn’t a ceiling set by what AI can do — it’s a floor set by the architecture they used. A pipeline with verification, scoped context, and quantitative grounding brought that to 84%.

The stakes are real. GiveWell has directed over $120 million to water chlorination programs alone. If a red teaming pipeline catches one issue that changes a cost-effectiveness estimate by 10%, that’s a meaningful input into decisions affecting millions of people.

Try It Yourself

The entire project is open source. The repo includes:

• An analysis of GiveWell’s current approach and its structural limitations
• A full architecture spec for the multi-agent pipeline
• All seven agent prompts, ready to use
• A manual workflow you can run in Claude with no code
• An automated Python pipeline for reproducibility
• Phase 1 results for water chlorination

Phase 1 covers water chlorination. I’m running the same pipeline against GiveWell’s ITN (bed net) and seasonal malaria chemoprevention analyses next — those have the strongest baselines to compare against since GiveWell published more detailed AI output for them. Results will be added to the repo as they’re completed.

If you work at GiveWell or a similar research organization, the manual version is designed for you — sequential prompts you can run today with zero engineering. If you’re a developer interested in agent architectures, the automated pipeline and the spreadsheet sensitivity analysis module might be more your speed.

github.com/tsondo/givewell_redteam

If you have questions, want to talk about the methodology, or want to point out something I got wrong, I’m at [email protected] or on BlueSky @tsondo.com. Or find me via links in the about.