2024 African Meeting, Abidjan, Côte d'Ivoire : June, 2024
Implementation Matters: Generalizing Treatment Effects in Education
Noam Angrist, Rachael Meager
Targeted instruction is one of the most effective educational interventions in low- and middle-income countries, yet reported impacts vary by an order of magnitude. We study this variation by aggregating evidence from prior randomized trials across five contexts, and use the results to inform a new randomized trial. We find two factors explain most of the heterogeneity in effects across contexts: the degree of implementation (intention-to-treat or treatment-on-the-treated) and program delivery model (teachers or volunteers). Accounting for these implementation factors yields high generalizability, with similar effect sizes across studies. Thus, reporting treatment-on-the-treated effects, a practice which remains limited, can enhance external validity. We also introduce a new Bayesian framework to formally incorporate implementation metrics into evidence aggregation. Results show targeted instruction delivers average learning gains of 0.42 SD when taken up and 0.85 SD when implemented with high fidelity. To investigate how implementation can be improved in future settings, we run a new randomized trial of a targeted instruction program in Botswana. Results demonstrate that implementation can be improved in the context of a scaling program with large causal effects on learning. While research on implementation has been limited to date, our findings and framework reveal its importance for impact evaluation and generalizability.
Preview
![Page 1](/event_papers/download/265/133/1/Implementation_Matters_p1.jpg)
![Page 2](/event_papers/download/265/133/1/Implementation_Matters_p2.jpg)
![Page 3](/event_papers/download/265/133/1/Implementation_Matters_p3.jpg)
![Page 4](/event_papers/download/265/133/1/Implementation_Matters_p4.jpg)
![Page 5](/event_papers/download/265/133/1/Implementation_Matters_p5.jpg)
![Page 6](/event_papers/download/265/133/1/Implementation_Matters_p6.jpg)
![Page 7](/event_papers/download/265/133/1/Implementation_Matters_p7.jpg)
![Page 8](/event_papers/download/265/133/1/Implementation_Matters_p8.jpg)
![Page 9](/event_papers/download/265/133/1/Implementation_Matters_p9.jpg)
![Page 10](/event_papers/download/265/133/1/Implementation_Matters_p10.jpg)
![Page 11](/event_papers/download/265/133/1/Implementation_Matters_p11.jpg)
![Page 12](/event_papers/download/265/133/1/Implementation_Matters_p12.jpg)
![Page 13](/event_papers/download/265/133/1/Implementation_Matters_p13.jpg)
![Page 14](/event_papers/download/265/133/1/Implementation_Matters_p14.jpg)
![Page 15](/event_papers/download/265/133/1/Implementation_Matters_p15.jpg)
![Page 16](/event_papers/download/265/133/1/Implementation_Matters_p16.jpg)
![Page 17](/event_papers/download/265/133/1/Implementation_Matters_p17.jpg)
![Page 18](/event_papers/download/265/133/1/Implementation_Matters_p18.jpg)
![Page 19](/event_papers/download/265/133/1/Implementation_Matters_p19.jpg)
![Page 20](/event_papers/download/265/133/1/Implementation_Matters_p20.jpg)
![Page 21](/event_papers/download/265/133/1/Implementation_Matters_p21.jpg)
![Page 22](/event_papers/download/265/133/1/Implementation_Matters_p22.jpg)
![Page 23](/event_papers/download/265/133/1/Implementation_Matters_p23.jpg)
![Page 24](/event_papers/download/265/133/1/Implementation_Matters_p24.jpg)
![Page 25](/event_papers/download/265/133/1/Implementation_Matters_p25.jpg)
![Page 26](/event_papers/download/265/133/1/Implementation_Matters_p26.jpg)
![Page 27](/event_papers/download/265/133/1/Implementation_Matters_p27.jpg)
![Page 28](/event_papers/download/265/133/1/Implementation_Matters_p28.jpg)
![Page 29](/event_papers/download/265/133/1/Implementation_Matters_p29.jpg)
![Page 30](/event_papers/download/265/133/1/Implementation_Matters_p30.jpg)
![Page 31](/event_papers/download/265/133/1/Implementation_Matters_p31.jpg)
![Page 32](/event_papers/download/265/133/1/Implementation_Matters_p32.jpg)
![Page 33](/event_papers/download/265/133/1/Implementation_Matters_p33.jpg)
![Page 34](/event_papers/download/265/133/1/Implementation_Matters_p34.jpg)
![Page 35](/event_papers/download/265/133/1/Implementation_Matters_p35.jpg)
![Page 36](/event_papers/download/265/133/1/Implementation_Matters_p36.jpg)
![Page 37](/event_papers/download/265/133/1/Implementation_Matters_p37.jpg)
![Page 38](/event_papers/download/265/133/1/Implementation_Matters_p38.jpg)
![Page 39](/event_papers/download/265/133/1/Implementation_Matters_p39.jpg)
![Page 40](/event_papers/download/265/133/1/Implementation_Matters_p40.jpg)
![Page 41](/event_papers/download/265/133/1/Implementation_Matters_p41.jpg)
![Page 42](/event_papers/download/265/133/1/Implementation_Matters_p42.jpg)
![Page 43](/event_papers/download/265/133/1/Implementation_Matters_p43.jpg)
![Page 44](/event_papers/download/265/133/1/Implementation_Matters_p44.jpg)
![Page 45](/event_papers/download/265/133/1/Implementation_Matters_p45.jpg)
![Page 46](/event_papers/download/265/133/1/Implementation_Matters_p46.jpg)
![Page 47](/event_papers/download/265/133/1/Implementation_Matters_p47.jpg)
![Page 48](/event_papers/download/265/133/1/Implementation_Matters_p48.jpg)
![Page 49](/event_papers/download/265/133/1/Implementation_Matters_p49.jpg)
![Page 50](/event_papers/download/265/133/1/Implementation_Matters_p50.jpg)
![Page 51](/event_papers/download/265/133/1/Implementation_Matters_p51.jpg)
![Page 52](/event_papers/download/265/133/1/Implementation_Matters_p52.jpg)
![Page 53](/event_papers/download/265/133/1/Implementation_Matters_p53.jpg)
![Page 54](/event_papers/download/265/133/1/Implementation_Matters_p54.jpg)
![Page 55](/event_papers/download/265/133/1/Implementation_Matters_p55.jpg)
![Page 56](/event_papers/download/265/133/1/Implementation_Matters_p56.jpg)
![Page 57](/event_papers/download/265/133/1/Implementation_Matters_p57.jpg)
![Page 58](/event_papers/download/265/133/1/Implementation_Matters_p58.jpg)