I Pointed Bayesian Optimization at GiveWell's Bed Net Model. Here's What It Found.
A few days ago I wrote about adapting Meta’s BOxCrete framework to optimize global health interventions. The premise: if Bayesian optimization can find better concrete mixes by navigating complex design spaces with competing objectives and noisy data, it should be able to find better configurations for malaria bed net programs. Same optimization structure, different domain.
I built it. The results taught me more about GiveWell’s model than about optimization.
Phase 1: The Model Is Too Simple
The first step was reimplementing GiveWell’s insecticide-treated net (ITN) cost-effectiveness analysis as a pure Python model. No spreadsheet dependency, fast enough to call thousands of times. I validated it against GiveWell’s actual spreadsheet across all eight country programs and got 0% error on baseline reproduction.
Then I pointed a Gaussian process surrogate with Bayesian optimization at five key parameters the red-teaming pipeline had identified as high-impact: the mortality incidence reduction estimate, a net usage adjustment, an external validity discount, an indirect deaths multiplier, and an over-5 relative efficacy factor. I ran per-country optimization in both directions (minimize cost per DALY for the best defensible case, maximize for the worst) across all eight programs.
Every parameter was monotonic. The optimizer pushed every input to its bound in every run. There were no trade-offs, no local optima, no parameter interactions. Cost per DALY turned out to be a simple multiplicative function where every factor independently pushes the output in one direction. A corner evaluation would have found the same answer.
This isn’t a failure of the optimizer. It’s a finding about the model. In concrete mix design, more cement helps strength but hurts workability. That tension is what makes optimization interesting. GiveWell’s CEA has no equivalent tension. Every “improvement” to any parameter independently improves the output.
The useful output was the uncertainty ranges. Running the optimizer in both directions maps the space of defensible assumptions:
| Country | Best Case CE | GiveWell Default | Worst Case CE |
|---|---|---|---|
| Guinea | 35.0x | 22.8x | 5.5x |
| Chad | 7.5x | 4.8x | 1.2x |
Guinea’s cost-effectiveness could plausibly be anywhere from 5.5x to 35x GiveWell’s benchmark. Chad ranges from just above break-even to 7.5x. Those ranges, derived from defensible parameter bounds, are something GiveWell’s point estimates don’t surface.
But the deeper finding is structural: the model lacks the internal complexity that would make optimization non-trivial. Real intervention design involves trade-offs the CEA cannot capture.
Phase 2: Adding One Realistic Feature Changes Everything
The red-teaming pipeline had flagged exactly this problem. Across three interventions, the same structural pattern appeared: key parameters are modeled as constants when the evidence says they change over time. For ITNs specifically, nets degrade. Physical integrity declines. Insecticide potency fades. Usage drops. AMF’s own post-distribution monitoring shows only 31% of nets in usable condition at 24 months, with usage declining from about 80% at distribution to 65% at one year.
GiveWell’s model treats net efficacy as constant over the implicit 2.5-year distribution cycle. That’s the static assumption the red-teaming pipeline identified. Phase 2 replaces it with reality.
I built a temporal extension that models three decay curves calibrated to field data: physical survival (exponential decay, calibrated so 31% survive at 24 months per AMF data), insecticide efficacy (faster decay, roughly 12-18 month half-life per WHO durability monitoring), and usage (slow decline, calibrated to AMF’s 80%-to-65% trajectory over 12 months). The combined efficacy at any point in time is the product of all three.
Then I added distribution frequency as a design variable. Instead of fixing the cycle at 30 months, the optimizer can choose any interval from 12 to 48 months. Each distribution event fully replaces old nets, resets efficacy to initial levels, and incurs both variable costs (nets themselves) and a fixed logistics overhead (supply chain mobilization, transport, community health worker coordination).
This creates a real trade-off. Shorter cycles mean higher average net efficacy but more logistics events. Longer cycles save on distribution costs but let efficacy decay further between replacements. The fixed logistics cost per cycle is the term that penalizes frequent redistribution, and it’s what gives the optimization surface genuine curvature.
What the Optimizer Found
Distribution frequency showed non-monotonic behavior. Five of eight countries landed at interior optima, meaning the optimizer found a genuine best interval rather than pushing to a bound. The remaining three hit the 48-month upper bound, indicating that for those countries (with lower malaria burden), even infrequent distribution is cost-effective enough that logistics savings dominate.
All eight countries converged to optimal intervals of 42-48 months. Every single one longer than GiveWell’s implicit 30-month cycle. The finding is consistent: under defensible decay assumptions, distributing nets less frequently and accepting lower average efficacy is more cost-effective than the current cycle, because each distribution campaign’s logistics costs outweigh the marginal efficacy gain from fresher nets.
I tested whether this result depends on the logistics cost assumption, which is the least empirically grounded parameter in the model. Sweeping fixed logistics fraction from 5% to 25% of variable cost across three representative countries, the optimal interval held steady in the low 40s across all assumptions. The country-level results are driven by decay parameter differences, not logistics cost assumptions. Dollar-per-DALY values shift about 19% across logistics assumptions at a fixed interval, but the qualitative finding doesn’t change.
The temporal model also revealed that GiveWell’s static CEA understates cost per DALY by approximately 2x on average, because it assumes constant net efficacy over the distribution cycle. When you integrate actual decay curves over time, the realized protection is substantially lower than what the static parameters imply.
What Didn’t Change
The original five CEA parameters remained fully monotonic even with temporal dynamics added. I had hoped that introducing time-varying efficacy would create cross-parameter interactions, where the optimal value of one parameter depends on the value of another. It didn’t. The model’s structural flatness runs deeper than the static-vs-dynamic distinction. Adding one source of realism created curvature in the new design variable (distribution frequency) but didn’t change the fundamental character of the existing parameters.
Insecticide decay rate showed hints of interaction with distribution frequency (3 of 16 runs landed at interior optima for that parameter), suggesting that richer degradation models might eventually create more complex optimization surfaces. But 3 of 16 is suggestive, not conclusive.
What This Means
Two concrete findings came out of this work:
First, GiveWell’s static CEA overstates ITN cost-effectiveness by roughly 2x because it assumes constant net efficacy over the distribution cycle. This is a quantified version of the qualitative critique the red-teaming pipeline identified: static parameters for dynamic phenomena produce systematic overestimation of multi-year benefits.
Second, the optimal distribution interval under defensible decay assumptions is 42-48 months, not the implicit 30 months. This suggests that distributing nets less frequently could actually improve cost-effectiveness, a counterintuitive finding driven by the logistics cost of each distribution campaign.
Both findings come with important caveats. The decay curves are calibrated to published field data but use simple exponential models. The logistics cost per cycle is modeled as a fraction of variable cost without detailed field data on actual campaign costs. And the model treats each distribution as full replacement, which is how mass campaigns work in practice but ignores household-level net management.
What I Learned About Applying Bayesian Optimization to CEA Models
The honest summary is that Bayesian optimization was the wrong tool for Phase 1 and earned its keep in Phase 2.
GiveWell’s CEA is a multiplicative chain. Every parameter independently pushes cost-effectiveness in one direction. There’s nothing for an optimizer to discover that a simple sensitivity analysis wouldn’t find. I could have run each parameter to its bounds independently and gotten the same uncertainty ranges without fitting a Gaussian process surrogate. Phase 1’s value was in revealing that structural property of the model, not in the optimization itself.
Phase 2 was different. Adding temporal dynamics and a design variable with competing cost-vs-efficacy pressures created a surface with genuine curvature. The optimizer found interior optima that wouldn’t be obvious from one-at-a-time parameter sweeps. That’s the kind of problem Bayesian optimization is built for, and it’s the kind of problem that emerges only when you move beyond static point estimates.
The broader lesson: optimization is only as interesting as the model it operates on. If the model lacks trade-offs, the optimizer will tell you so. That’s useful information, but it’s information about the model, not a result of the optimization. The productive direction is making the model more realistic until the optimization surface becomes non-trivial, then using the optimizer to map the terrain.
What’s Next
The temporal extension currently covers ITNs only. Water chlorination has analogous dynamics (adherence decay, seasonal turbidity variation) and seasonal malaria chemoprevention has its own (drug resistance accumulation across treatment cycles). Each would require intervention-specific decay models but could use the same optimization framework.
The code and all results are open source in the same repo as the red-teaming pipeline.
github.com/tsondo/givewell_redteam
Same channels: [email protected] · BlueSky @tsondo.com · or via the about page.