The Textbooks are Still Too Hard

Last month, In Development magazine reprinted an old interview with Michael Kremer, in which he discusses the genesis of one of the first randomised controlled trials (RCTs) in development economics.

I had taught in a school in Kenya, and seeing how few resources the schools had made me think it must be good to have more resources, including more textbooks. But we found that textbooks only improved test scores for those who were already performing well. That made me think about the issues of fit between the curriculum and where the students currently were.

Textbooks were written at a level far from the level of many students. One of the big lessons from RCTs in education is the mismatch between where curricula are and where teaching is oriented and where the typical student is.

That insight is now reasonably well understood amongst development economists, from Kenya, where textbooks helped only the strongest pupils, to studies showing curricula running ahead of what children know. But as far as I know it hasn’t been shown systematically across countries. As in, can you show directly that textbooks across low- and middle-income countries are too hard for most children?

The most obvious answer is “most of the children can’t read, so of course they can’t read the textbooks”. Which has some truth to it, but reading isn’t a binary; it’s a series of steps. In principle, a child might be able to fail a reading test but still be able to follow a simpler book. Well, I have a corpus of textbooks and a Claude subscription, so it’s now reasonably straightforward to compare the difficulty of the tests that students sit with the books they’re expected to read in class.

Concretely, I found 433 primary school books from 11 countries that also have public data from nationally representative learning assessments. A lot of lesson time is spent on these books, so they matter. For each country, I compare extracts from the books with test questions from the grade above. So, for example, a child taking a grade 5 test could reasonably be expected to read a grade 4 textbook. If most children fail the grade 5 test, and the grade 4 book is no harder than the grade 5 test, we can conclude the grade 4 books are still too hard.

The bottom line is that nearly all of the books are harder than the test. The test pass rate gives us a ceiling on the share of kids who might be expected to read the books. For the 11 countries we have data from, Kenya—where Kremer did his original RCT in the 1990s—actually comes out best, with up to 75 percent of children now expected to be able to read their textbook (albeit with a small sample of just five books). Worst are India (23 percent) and Malawi (17 percent). This all adds up to millions of children sitting in classrooms spending hours staring at a textbook they can’t read. You can read the method details on GitHub here.

Figure 1. In most countries, fewer than half of children could read their textbooks

The most children who could read a textbook picked at random, grades 1–6

Image

Note: An AI judge (Jev) compared each textbook with the story in the country’s reading test. If the book was harder, we assume only children who passed the test, measured in the grade after the book’s, could read it. If it was no harder, we assume every child could. Bars are 90% intervals from the number of books and the survey’s sampling error. Jev is TypeSafe’s AI model for comparing texts; how it is used is set out at github.com/Center-for-Global-Development/textbooks-too-hard. Countries with at least five scored books are shown. Reading data are from MICS6, ASER (India, rural; Pakistan), Uwezo (Kenya, Tanzania, and Uganda), and PIRLS 2021 (South Africa).

Even the books for the youngest children are too hard. Sixty-one of the 63 grade 1 books and 44 of the 47 grade 2 books are harder than their country’s test story, which is pitched at about a grade 2 level, and which most grade 3 children fail (in every country except Guyana).

None of this is new. Pritchett and Beatty showed years ago that the mismatch between curricula and student ability means you could get more learning with easier materials. But “dumbing down” curricula is a hard sell to policymakers, as our recent survey with policymakers shows. The better solution is helping children catch up.