Evaluating learning and enrollment gains for the Educate Girls Development Impact Bond

IDinsight conducted a rigorous three-year impact evaluation of Educate Girls’ program in Rajasthan’s Bhilwara district to determine outcome payments in the Educate Girls Development Impact Bond (EG DIB).

Educate Girls surpassed the DIB targets for both learning gains and enrollment. Students in program villages gained an additional 8,940 ASER learning levels relative to students in control villages, surpassing the learning target set by the Development Impact Bond by 60%. The effects of Educate Girls’ program on learning gains were large and statistically significant over the three-year program: Students in EG schools gained on average an additional 1.08 learning levels, or 28% and 0.31 standard deviations, compared to students in control schools. By the end of the three-year project, Educate Girls had enrolled 768 out-of-school girls, representing 92% of all identified out-of-school school girls eligible for enrollment. Educate Girls thus exceeded the enrollment target of 79% by 16%.

Read the full technical report

Client: Children’s Investment Fund Foundation (CIFF)
Partners: Educate Girls, Instiglio, UBS Optimus Foundation
Location: India
Sector: Education
Dates of service: 2012 – 2018
IDinsight service: Randomized evaluation
IDinsight contacts: Jeffery McManus, Kate Sturla

Background – Educate Girls mobilizing communities for girls’ education in rural India

Nearly three million girls, aged 6 to 14, are out of school in India.[1] Even if enrolled in school, many students are not acquiring foundational skills like reading and basic arithmetic that can help them progress in school and life.[2]

Educate Girls, a Mumbai-based NGO, trains community volunteers to encourage families to enroll their girls in school by making door-to-door household visits and to deliver a child-friendly supplementary curriculum in classrooms to improve basic reading and math skills. Educate Girls currently operates in over 12,000 villages and 21,000 primary schools across 15 districts in Rajasthan and Madhya Pradesh.[3]

The Educate Girls Development Impact Bond – Testing a new tool for financing social programs

A Development Impact Bond (DIB) has been set up to finance Educate Girls’ program in Bhilwara district, Rajasthan. DIBs are a new tool for financing social programs in which an investor provides capital up-front to a service provider and earn a return from a donor based on the effectiveness of the program. By shifting the focus from inputs to outcomes and incentivizing service providers to improve programs, DIBs hold the potential to make social programs more cost-effective. The lessons from this world’s first DIB are informing the viability and design of future results-based financing models.

The Educate Girls Development Impact Bond (EG DIB) is a joint project between the Children’s Investment Fund Foundation (CIFF), Educate Girls, the UBS Optimus Foundation, Instiglio, and IDinsight (collectively, the “Working Group”). UBS Optimus, acting as the investor, financed EG’s project implementation, while CIFF paid for educational outcomes as evaluated by IDinsight. Instiglio is managing the project.

IDinsights service – Rigorous evaluation of learning and enrollment to determine DIB payments

IDinsight designed and conducted a three-year impact evaluation of Educate Girls’ program in Rajasthan to determine financial returns in the EG DIB. We measured two outcomes: learning gains of boys and girls in grades 3-5 and enrollment of out-of-school girls.

Learning gains, which made up approximately 80% of the final DIB payments, were evaluated in a village-clustered randomized control trial, the gold standard of scientific evidence. The evaluation involved a sample of ~12,000 students across 332 schools in 282 villages. Half of these villages were randomly assigned to receive Educate Girls’ program while the other half represented the comparison group. Learning gains were calculated as an aggregate treatment effect (instead of an average treatment effect) to incentivize Educate Girls helping low performers stay in school.

We assessed students on basic literacy and math competencies using the Annual Status of Education Report (ASER) testing tool; a student’s score on ASER determined her “learning level,” which is scored out of 16 points and forms the basis of the learning metric. Student assessments were conducted at the beginning of the project (baseline) and the end of the three following school years. IDinsight teams visited both treatment and control schools in the same geographic cluster on the same day. If students were not present at school on the day of the assessment, our surveyors assessed them at their homes.

Enrollment of out-of-school girls, which contributed the remaining ~20% of the payments, was measured using a simple pre-post evaluation in the 141 villages where treatment schools are located. Due to the cost of conducting a census of all households, the project partners decided against estimating enrollments in control villages. Thus, unlike the learning estimates, the enrollment estimates do not reflect a causal effect of EG’s program.

The enrollment outcome was calculated as percent of out-of-school girls enrolled out of all eligible out-of-school girls identified in the program villages by the end of the program. At the beginning of each school year, Educate Girls submitted a list of all out-of-school girls ages 7-14 living in treatment villages. IDinsight validated that list by visiting a sample of girls in the area and verifying their eligibility. At the end of each school year, Educate Girls reported successful enrollments and IDinsight validated all enrollments by visiting the schools and verifying the enrollment in the school register.

In addition to determining final payments, midline results from IDinsight’s evaluation enabled Educate Girls to fully realize the flexibility in programming provided by DIB set-up and make effective, data-driven course corrections along the way.

IDinsight submitted a pre-analysis plan for the evaluation at the Registry for International Development Impact Evaluations (RIDIE). For more information on the evaluation design, please refer to the pre-analysis plan or the DIB design document.

 

Mandalgarh_1Sept2015_1-1024x768

Ramavatar Dhonli, a surveyor, records a child’s responses to the ASER learning assessment on a mobile phone during piloting for the EG DIB in September 2015. (Photo credit: Kate Sturla)

Additional resources

About IDinsight’s evaluation design and process: http://bit.ly/2badJQe

Registered pre-analysis plan: http://ridie.3ieimpact.org/index.php?r=search/detailView&id=369

[1] Source: http://www.educategirls.in/What-We-Do.aspx#current-situation

[2]Source: http://img.asercentre.org/docs/Publications/ASER%20Reports/ASER%202016/Report%20sections/aser2016-nationalfindings.pdf

[3]Source: https://www.educategirls.ngo/pdf/English%20-%20Information%20Brochur.pdf