Measuring the impact of interventions that positively affect the livelihoods of people is the core of Laterite’s business. Methodologically, the golden standard for this in the bio-medical sciences is randomized controlled trials (RCTs). However, implementing them in the context of social interventions poses additional challenges than in the controlled environment of the bio-medical setting. In this blog, we explore some of the challenges we faced in implementing an RCT in an agroforestry intervention in Rwanda and what we did to resolve them.
Founded in 2006, One Acre Fund (1AF) is a non-profit social enterprise present in the rural areas of nine African countries, dedicated to making smallholder farmers more productive and resilient. In Rwanda, the 1AF agroforestry program provides 1.4 million farmers with tree seedlings. In 2023, around 90% of the program will be designed as a decentralized nursery model, where one single nursery will be set up in a central location in a cell (smallest administrative unit in Rwanda). A nursery manager from the community will nurture the seeds until the seedling stage and will be paid for each healthy tree seedling produced. Around three months after planting, farmers of each cell will come to the nursery to pick up the seedlings for free.
We intend to measure the impact of the program in several outcomes, mainly tree planting, tree survival and the difference between the trees that survived in a treatment and a control group. This aims to quantify the number of additional trees that have been planted and survived due to the program implementation. This will be used to estimate tree value and the financial impact of the intervention.
The map below shows excluded districts in dark gray, and buffer cells surrounding them and national borders in light gray.
Here we highlight several challenges we faced in the study and how we went about solving them:
- Most of the country already receives treatment – From the beginning, this case differs from a traditional RCT since most cells are already being offered the agroforestry program. Instead of selecting control areas, we needed to choose cells where the program will be temporarily stopped. Farmers in control cells would be compensated by receiving fruit trees.
- Presence of other organizations delivering similar interventions – In three districts of Rwanda (Gicumbi, Rutsiro and Rulindo), other organizations comparable to 1AF had also established a presence. Measuring outcomes in these areas were likely to be impacted by this and would provide a biased estimate of the impact of the intervention. We excluded these three districts from the sampling frame, in addition to creating a buffer area in the cells surrounding these provinces to ensure we exclude areas affected by spillovers.
- Cells next to international borders present additional challenges – These include issues surrounding security and logistics, as well as potentially being different in terms of challenges and market limitations that farmers face. We decided it was best to exclude them from the sampling frame. Approximately 8% of the cells were excluded.
- Different species of trees are provided depending on area – Although grevillea trees are by far the most distributed species of tree by 1AF, they also distribute other species, and this varies within Rwanda. To include a set representative of this variation, we created stratification areas that reflected this variation and sampled accordingly.
We implemented this RCT in the context of a well-established program where other stakeholders operate and where there are nuances in the selection of areas. These prove a challenge for impact measurement, but are often the conditions we face in the evaluation of programs aiming to improve the livelihoods of people in the countries where we work. At Laterite, we collaborate with our clients to provide technically sound approaches to doing so.
This blog post was written by Jose Rubio-Valverde, Associate Economist and Judith Bayer, Senior Research Associate at Laterite.