American Evaluation Association
October 27-31, 2013
Presenter: Rebecca Firestone
Session Title: Coarsened Exact Matching: Methods and Application for Evaluating Global Health Programs
Session Chair: Rebecca Firestone
This session describes the result of a partnership between the Institute for Quantitative Social Sciences at Harvard University and Population Services International (PSI), a global health NGO, to test the use of a statistical matching method, coarsened exact matching (CEM), for application to PSI’s needs to estimate attributable impact of its social marketing and service delivery programs. CEM is designed to reduce imbalance between treatment and control groups derived from observational data, while reducing model dependence and error in estimating causal effects. The procedure is easy to implement. We show that it can maximize use of available sample size compared to other matching procedures. The first paper of this session describes the coarsened exact matching procedure and its attributes. The second paper describes an application of the CEM procedure in a study evaluating the impact of a campaign to increase HIV risk perceptions around concurrent sexual partnerships in Botswana.
Presentation Title: Evaluating Botswana’s Campaign on Concurrent Partnerships using Coarsened Exact Matching
Authors: Noah Taruberekera, Iris Halldorsdottir, Rebecca Firestone, Lung Vu, Virgile Capo-Chichi
The evidence linking concurrent sexual partnerships (CP) to HIV transmission prompted the Government of Botswana to launch a national campaign in 2009, with PSI as an implementing partner. This study was designed to evaluate the impact of the campaign on reducing CP among adults aged 18-35. We hypothesized that the intervention increased factors protective against CP, including risk perceptions, social norms, and perceived benefits of monogamous relationships, and reduced CP. A national cross-sectional survey was conducted in 2011 using a multi-stage sampling design. Individuals were randomly sampled from households within enumeration areas. Coarsened exact matching was used to create statistically equivalent groups of treated and non-treated individuals, based on radio/TV ownership and place of residence. Logistic regression was used among the matched sub-sample to assess treatment effects. Exposure to campaign messages was associated with increased risk perceptions regarding CP (OR=1.49; 95%CI=1.11-2.01), condom use self-efficacy (OR=1.38;95%CI=1.02-1.88), and HIV testing (OR=1.6; 95%CI=1.06-2.42).
Coarsened Exact Matching: Methods and Application for Evaluating Global Health Programs
The need for rigorous impact evaluation of global health programs has become paramount in a changed funding environment that demands accountability and measurable results. However, increased rigor in evaluation studies is often challenging to marry with programmatic needs for rapidly produced results and adaptability to changing implementation plans.
The Institute for Quantitative Social Sciences at Harvard University and Population Services International (PSI) have partnered on testing the use of a new matching technique, coarsened exact matching (CEM), for application to PSI’s evaluation needs. As a global health NGO working in HIV/AIDS, reproductive health, malaria and child survival, PSI requires evaluation methods to validly estimate attributable impact, i.e. average treatment effects, of its social marketing and service delivery initiatives. The use of statistical matching provides an important opportunity to improve PSI’s ability to make causal inferences regarding the impact of its programs.
Statistical matching preprocesses observational data to account for some or all of the risk of selection bias from pretreatment variables that influence the likelihood of receiving a treatment, thereby reducing imbalance between treatment and control, i.e. non-treated, groups. CEM is a Monotonoic Imbalance Bounding matching method where the balance between the treated and control groups is chosen by the user ex ante. CEM strictly bounds model dependence and the error in estimating average treatment effects through. With CEM, users temporarily coarsen, or categorize, their data, exact match on the coarsened date, andt hen run subsequent analyses on the uncoarsened, matched data. CEM is robust to measurement error, works well with multiple imputation methods for missing data, is less likely to limit sample size than other matching procedures, and is simple to implement, even for data analysts with limited experience.
This session aims to introduce coarsened exact matching to the evaluation community and discuss how the method can be used in practice. Attendees will become familiar with a flexible and robust quantitative method for improving causal inference and one which can be used by field organizations seeking computationally light, but still rigorous methods.
The first paper of this session describes the coarsened exact matching procedure and its specific attributes. The second paper describes an application of the CEM procedure to a study evaluating the impact of a campaign designed to increase HIV risk perceptions around concurrent sexual partnerships in Botswana.
- Populations Served
- General Population, Men at High Risk for HIV, Women at High Risk for HIV
- Health Areas
- HIV and Sexually Transmitted Infections
- Marketing Products and Services
- Resource Types
- Condoms and Lubricant, HIV Prevention, Male Condom