What Works in Edtech?

With many schools around the world still closed due to COVID-19, governments are searching for ways to ensure continuity of learning. Edtech obviously has its limits as a replacement for school during pandemic-response closures: Many households in poor countries don't yet have reliable electricity or internet. And despite recent growth, few edtech firms have yet reached any substantial scale.

But for governments that do want to invest in edtech, where should they start?

In a recent working paper, one of us (Dan) summarised all 67 experimental or quasi-experimental studies to date on edtech in developing countries. The studies fall into four broad categories:

1. Self-led learning software
2. Improvements to instruction
3. Access to technology
4. Technology-enabled behavioral interventions

In this blog, we focus on the household-based interventions that could be most impactful during the current crisis.

Self-led products can help

Self-led learning software is one of the most promising areas of research. Products studied have consistent medium-to-large effects on learning (Figure 1). An important feature of these products is adaptation to each student’s level. This allows students to progress at their own pace and with material that is relevant for them. This kind of differentiation is important at the best of times, and even more so after differential learning loss from school closures.

Figure 1. Effect sizes of self-led learning programs

Image

Notes: "learning gain” coded as the largest gain in any field of learning, whether it is an academic subject like “math” or a less-established area like “computer skills.” Studies for which authors did not report enough information to standardize gains into standard deviation (SD) units are not in this plot. Studies denoted with a star (*) did not report enough cost information to obtain a per-pupil estimate. Studies without a confidence interval did not report standard errors.

Many studies on self-led learning software were though tested on school-owned or other shared hardware. This won't be much use whilst access to shared facilities is restricted. In many places, the basics required to do edtech at home are missing: 175 million people don't have electricity at home in Nigeria, Ethiopia, and Democratic Republic of Congo. Even where such prerequisites do exist, inequality in access can be stark. In Mexico and Peru, almost all the richest households have computers at home. For the poorest 20 percent of households, less than one in ten do. One obvious potential solution is focusing on technologies with high penetration such as SMS, feature phones, or radio.

Finally, although the self-led learning products that have been rigorously tested are promising, they are far from a representative sample of all products. Many have yet to be rigorously tested, and there is likely huge variation in quality.

One-Laptop-Per-Child?

If students don't have hardware at home, should governments provide it? The current evidence shows that distributing devices alone is not an effective strategy. This is best exemplified by disappointing results of the “one-laptop-per-child” initiative across five Latin American countries (for example, Peru, Uruguay, or Colombia). Access to hardware might be necessary for access to effective software, but it certainly isn't sufficient, and is unlikely to be the most cost-effective use of public money.

In the right circumstances, remote instruction is possible

Remote instruction has been hastily launched around the world during school lockdowns. Radio instruction has been tried in many contexts before the current crisis (and was in fact one of the first ever randomized controlled trials in the field of development). More recent technologies have been tried in Pakistan, India, Ghana, China, and Paraguay. Approaches have included pre-recorded lessons, live streaming, supplementing in-person instruction, or replacing it entirely. All have proved effective at raising learning outcomes.

Adapting Sesame Street, an American educational TV show, for local audiences has been shown to improve foundational skills in Indonesia and India. Tanzanian educational cartoon series Ubongo has seen similar results in Tanzania and Rwanda. While most of these interventions were tested at the school level, the promising effects and large potential for scalability to reach more children in the rest of the respective countries makes them strong candidates for distance learning if school closures due to COVID continue.

Edtech can help with student engagement, post-closure

Beyond short-term losses in learning, the current crisis will also lead to some children dropping out of school altogether, which could revert a large portion of the recent gains in enrollment across the world.

Behavioral interventions around the world have proven effective at overcoming behavioral and informational barriers. Simple phone calls promoting accountability helped adult learners in Niger, and texts to parents about their children’s performance and attendance improved schooling outcomes in Chile. The scalability and reach of this type of intervention is a promising avenue to encourage parents and students alike to engage with education during the lockdown, and to return to school after it.

Edtech is a valuable option for educational systems, but not the only one

Given the costs and barriers to entry, it isn't clear that edtech interventions will always achieve higher learning gains than other interventions. There are plenty of non-tech interventions that can produce impressive learning gains. What really matters is cost-effectiveness and scalability.

Edtech works best when it addresses a clear binding constraint in the system where it is deployed. Different edtech solutions can be used as complements in an educational system, as the potential for scalability, goals, and populations differ drastically by intervention. Furthermore, edtech products should focus on the clear comparative advantage of technology—the potential to provide high levels of customization and individualization of practice and instruction. But policymakers should keep in mind the short-term limitations in terms of infrastructure that could hamper edtech’s potential for penetration. Limited access to electricity, internet, and even feature phones in the most remote areas could be dealbreakers for even the most innovative edtech product.

Overall the evidence from developing countries differs from that of rich countries (as summarized by Escueta et al.) in that the kinds of technologies available to policymakers and families to adapt to situations like the current moment are strikingly different. The type of edtech intervention that can be deployed in developing countries needs to be particularly sensitive to costs, scalability, and reach for the most disenfranchised populations in the world. Still, well-designed edtech solutions can be a promising way to minimize the ripple effects in education during and after school closures.

For a list of all coded papers, please see this document.

Has your paper not been included in the review? Please reach out to [email protected].