New Data on Learning Loss in Pakistan

The research is clear: children in rich countries learned less than usual last year, even with greater advantages in access to the internet, books, and well-educated parents at home. But we still know little about the effects of the COVID-19 pandemic and school shutdowns on education in developing countries, which have primarily relied on lower-tech solutions for distance learning such as radio or TV.

Estimates that used summer learning loss as a model for what might happen over a full year of education disruption predicted not just less progress than in a typical year but an absolute reduction in learning levels relative to how much children knew before the pandemic began. Some emerging data shows that children learned only 30-40 percent as much as a normal year in Ethiopia, and lost 3.5 months of learning in Kenya.

To get a better picture of the effects of the pandemic on education in Pakistan, we carried out another round of our survey of students and parents. We found gender differences in learning loss, little engagement with government teleschool, dropping parental support for further closures, and more.

The sample: A large private school network

Private schools serve a substantial portion of school-going children in Pakistan—39 percent of primary school students and 38 percent of middle school students in 2016. Across three rounds of surveys, we interviewed 1,528 households^[1] with children enrolled in both private and public-private partnership schools run by The Citizens Foundation (TCF). TCF is one of the largest charitable private school networks in the world—running 1,652 school units and educating 266,000 children across Pakistan. Donors heavily subsidise the schools, with fees to parents ranging from 1 percent to 30 percent of costs, depending on family income. The average student wealth in our sample is similar to the national average (albeit slightly poorer) as measured by asset wealth in the 2019 Pakistan Social and Living Standards Measurements, which assesses wealth based on possessions such as a car, a refrigerator, a fan, a stove, a motorcycle, the number of rooms in the household, type of drinking water, and toilet facility type. Seventeen percent of students in our sample are in the poorest quintile of all households at the national level and 10 percent are from the richest quintile.

Our first survey round (discussed in this blog) coincided with the peak of Pakistan’s first COVID-19 wave in June 2020. We conducted a second survey round in the trough between waves in September and just before schools reopened the first time. Schools reopened briefly on 23rd September 2020, and then closed again on 26th November because of rising cases. Our third survey followed the second wave, taking place after schools had reopened in February 2021. Cases in Pakistan are now rising again and some school closures in COVID-19 hotspots have been imposed for grades 1 to 8.

Figure 1. COVID and School Closures in Pakistan

Image

Note: Data on cases from OurWorldinData. Areas shaded in gray show the school closure periods. Data collection dates are shown on the x-axis.

Most children increased their time spent studying over time

Overall, there was an increase in children’s time spent studying from the peak of the first wave of cases in June 2020, to three months later in September when cases had come back down. The share of children whose primary activity reported by parents was studying increased from 40 percent to 60 percent. Time spent studying increased by around half an hour per day. There is also a difference by gender that is consistent with the learning results. Girls consistently spent more time than boys on studying across the three survey rounds: in round one, girls studied on average for 1.6 hours per day while boys studied for 1.5 hours. In round three, girls studied for 2.2 hours and boys for 1.9 hours. Boys also spent more time working outside of the house: this work was a main activity for 9 percent of boys (versus just 0.5 percent of girls) in September 2020, increasing to 15 percent of boys by February 2021 (compared to 0.6 percent of girls).

Figure 2. Most children increased their studying over time

Percent of children whose primary activity is studying and average daily hours spent with education activities at home

Image

Note: Sample size of 851 girls and 936 boys in round 1, 600 girls and 655 boys in round 2 and 717 girls and 769 boys in round 3. Areas shaded gray show school closure periods. Vertical bars in the indicate 95% confidence intervals. T-tests indicate boys average hours spent in education in Feb-2021 is statistically significant from their time spent in Sep-2020.

Though limited, our data suggest boys “lost” learning, while girls made some progress

We carried out a simple two-item student mathematics assessment by phone—a subtraction problem (“What is 39 minus 16?”) and a division problem ( “What is 48 divided by 2?”). We asked these two questions in September 2020 and February 2021. The data suggests that between the two rounds boys became less likely to answer correctly while girls became more likely to give the correct answer (Figure 2). While this analysis is based on a very short two-item assessment, the pattern is consistent with the change in average hours girls and boys spent studying between September and February.

How many children would we expect to learn these skills in a normal school year? We use data from an ASER maths assessment carried out in 2018 in Pakistan for children between 8 and 13 years old, to estimate normal progress across grades for children the same age in our sample. Specifically, we calculate the average difference in probability of being able to answer the same questions, by student grade, gender and wealth. This allows us to simulate what progress we might have expected if schools had remained open. This shows that girls remain roughly on track with expected progress, but boys might be dropping behind (Figure 3).

Figure 3. Learning loss for boys but not for girls

Image

Note: Sample restricted to children between 8 and 13 years old. Sample size of 165 girls and 152 boys in Sep. 2020 and 88 girls and 95 boys in Feb. 2021. Vertical bars indicate 95 percent confidence intervals. Note that February data indicates the same date, and is spaced out on the x-axis to ease visibility. Expected learning is estimated using cross-grade differences in ASER test results for children between the ages of 8 and 13 prior to the pandemic and school closures.

It also shows that children from the poorest households are falling behind in maths relative to children from the richest households and are not on track with learning subtraction operations as would be expected in usual times (Figure 4).

Figure 4. Learning loss for the poorest children for subtraction items

Image

Note: Sample size of 124 children in the poorest quartile and 144 in the wealthiest quartile in Sep. 2020 and 75 and 72 children in the poorest and wealthiest quartiles, respectively, in Feb. 2021. Vertical bars indicate 95 percent confidence intervals. Note that February data indicates the same date, and is spaced out on the x-axis to ease visibility. Expected learning is estimated using cross-grade differences in ASER test results for children between the ages of 8 and 13 prior to the pandemic and school closures.

Less than 1 in 4 watched government teleschool

A month after schools first closed, the government of Pakistan launched a teleschool program in April 2020. The television channel broadcasted free educational content for children in grades 1 through 12. Over 60 percent of our sample had a TV at home, but only a third of households with a TV reported watching teleschool. Considering the full sample (which includes people who don’t have access to a TV), only 22 percent of households watched teleschool.

Children with the most educated parents were 37 percent more likely to watch the TV channel than children with uneducated parents, after 3 months of closures. The most affluent families were 86 percent more likely to be watching than the poorest. These gaps grew over time, with more educated households becoming more likely to watch teleschool and the least educated and poorest less likely, after 6 months of school closure.

Figure 5. Richer families were twice as likely to watch government teleschool, and the gap grew over time

Percent of households watching teleschool, by parent education and asset wealth (%)

Image

Note: Data comes from survey round 1 and 2 in June and September 2020. Sample size of 1,208 respondents in round 1 and 892 respondents in round 2 for the education chart on the left. Sample size of 926 respondents in round 1 and 865 respondents in round 2 for wealth chart on the right.

In addition to government teleschool, TCF sent out learning materials to children, including magazines that also had information on audio resources supplementing the written material. TCF also aired episodes of their own in the government’s teleschool. Sixty percent of children engaged with the TCF magazine, and 18 percent watched at least one episode of TCF’s TV episodes. Over a third did not engage with either.

Engagement with TCF Distance Learning materials

Note: The sample included in this table is the eldest children in households who were available and parents consented to speak at the time of interview in round 3, February 2021. Thirty percent of our sample consented for children to speak. Because of attrition bias, there is an underrepresentation of the poorest families in this table.

In the final round only we also asked a spelling question (spell the word ‘job’) and a vocabulary question (name all the animals that you know). Forty-nine percent of children could answer the spelling question and the average vocabulary retrieval was seven words.

Looking at correlates of performance, we do see better scores for children who engaged more with TCF’s TV episodes and magazine. Children who watched an additional episode of TCF episodes scored 0.15 standard deviations better on the mathematics questions. Children who spent an additional hour with the TCF magazine scored 0.06 standard deviations better on both the mathematics and language questions. These are correlations and not necessarily causal relationships, so should be interpreted with caution, but the relationships are at least robust to controlling for children’s age, sex, and their parent’s level of education and wealth.

Figure 6. Some weak correlation between engagement with TCF materials and learning

Image

Note: sample size of 562 children in Sep 2020 and 314 children in Feb 2021. Test scores are calculated as the first principal component of the two test items for each subject, standardised to mean zero and standard deviation of one.

After 10 months of school closures, 86 percent of children returned to TCF schools and parental support for further closures dropped

Schools closed in March 2020 and remained closed until January 2021. Eighty-six percent of the children in our sample who were enrolled in school in June 2020 had re-enrolled in their TCF school in January 2021. Some of those who did not return to their TCF school did likely re-enrol elsewhere, but overall the most common reason given for not re-enrolling was that the family could no longer afford to send their child to school. Actual re-enrolment levels were similar to the stated intentions of parents in September 2020 and much lower than stated intentions in June 2020. The school year starts in April in Pakistan, meaning that though children re-enrolled, they had missed close to a full year of schooling.

Figure 7. Eighty-six percent of children returned to TCF schools

Intended and actual re-enrolment rates for girls and boys (%)

Image

Note: Areas shaded gray show school closure periods. Bars represent intentions to re-enrol girls and boys during school closures, and (*) actual re-enrolment rates as schools reopened.

Parent support for school closures dropped from 49 percent in June 2020 to 31 percent in January 2021. This is consistent across respondents of different educational and wealth backgrounds. More affluent and better educated families are more likely to support future school closures. This raises challenges for containment of COVID spread in Pakistan, as cases remain high.

Through three rounds of surveys spanning nine months of the pandemic, we’ve been able to get a unique look at the way that COVID has upended education in Pakistan and the significant and evolving gender and wealth gaps in learning. Look out here for updates from our partners in Ghana, Kenya, Malawi, Pakistan, and Senegal in the coming weeks and months as they produce new data on the education challenges posed by COVID-19.

Thanks to TCF and CERP for assistance with the sample and data collection for these surveys, and to Maryam Akmal for leading the survey design.

Thanks to Alexis Le Nestour, Maryam Akmal, and Hina Muhammad Saleem for helpful comments.

The survey data is available to download at this link.