Improving the enrollment process of a social cash transfer program
Client: World Food Program & Ministry of Community Development Mother and Child Health
Sector: Livelihoods, Survey Techniques & Measurement
IDinsight services: Impact evaluation, process evaluation
Cash transfer programs have been shown to be effective at improving welfare outcomes for the most vulnerable communities in Zambia (Seidenfeld, Handa, and Tembo 2013). Since 2004, the Government of the Republic of Zambia’s (GRZ) Social Cash Transfer Program (SCT) has offered cash assistance to Zambia’s poorest families. The SCT program, run by the Ministry of Community Development and Mother and Child Health (MCDMCH), is currently operational in 19 districts and will be extended to 50 districts by the end of 2014 and to all districts nationwide by 2016.
This expansion of SCT benefits provides an opportunity to improve the program’s enrollment process, which, to date, has been labor-intensive and vulnerable to error. In anticipation of this scale-up, various donors and partners are undertaking activities to support the MCDMCH in the expansion. One of these partners, the World Food Program (WFP), seeks to examine how a transition to electronic data collection methods (“m-tech”) could affect the enrollment process. WFP has engaged IDinsight to conduct an evaluation to determine which method of enumeration and registration system (paper versus m-tech) is fast, accurate, affordable, intelligible, and scalable.
To answer this question, the MCDMCH has agreed to allow a pilot of electronic enumeration for the SCT program in 3 districts in Northern and Lulupula provinces: Mungwi, Mwense, and Shiwangandu. Further, the Ministry has authorized WFP to contract IDinsight to carry out a quantitative evaluation and a process evaluation of the differences in error rate, time, costs and overall process of the paper and m-tech enumeration approaches.
The intervention is a comparison of the paper based system of enumeration and registration for the Ministry of Community Development Mother and Child Health’s Social Cash transfer program.
IDinsight is offering quantitative, qualitative, and process evaluation services as a part of this project.
For the quantitative evaluation, IDinsight used a matched pair approach to measuring difference in performance with the m-tech and paper enumeration methodologies. Each enumerator’s performance with paper surveys will be compared to his/her own personal performance with m-tech. This matched pair approach allows us to accurately measure the underlying difference in speed and accuracy between the two methods, without any confounding effects of inherent enumerator skill, most of which are intangible and difficult to measure and control for in analysis.
To capture enumerators’ performance with the m-tech and paper surveys, each enumerator will enumerate half of their assigned households with the m-tech and the other half working with the paper. To account for natural improvements in speed and accuracy that will take place over time, we will randomly assign enumerators to start with either the paper or the m-tech.
To calculate error rates, enumerator interviews are recorded and transcribed, then the transcribed information is compared to the data collected on paper and electronic forms. To measure corrections that take place at different stages of the enumeration process, photographs are taken of paper surveys immediately after they are completed, as well as after supervisors review and correct them. Similarly, the meta-data from the m-tech will indicate the number of time m-tech forms are modified or corrected.
Data for the process evaluation is collected through a series of qualitative and quantitative measures. Skills with the tablets are measured at multiple points during the enumeration process with a “Mobile Experience Survey,” and challenges faced in the field are captured in a “Mobile Challenges Survey” both conducted with enumerators. Additionally, qualitative interviews with enumerators, potential beneficiaries, district officials, provincial officials, and other stake holders contribute to the process evaluation.