Why is there no interest in kinky learning?

Just *how* poor are *your* beneficiaries though? In the aid project business everybody is obsessed with reaching the *poorest* of the poor. The ultra poor. The extreme poor. Lant Pritchett has criticised extensively this arbitrary focus on getting people above a certain threshold, as if the people earning $1.91 a day (just above the international poverty line) really have substantively better lives than those on $1.89 (just below). Instead he argues we should be focusing on economic growth and lifting the whole distribution, with perhaps a much higher global poverty line to aim at of around $10-15 a day, roughly the poverty line in rich countries.

Weirdly, we have the opposite problem in global education, where it is impossible to get people to focus on small incremental gains for those at the bottom of the learning distribution. Luis Crouch gave a great talk at a RISE event in Oxford yesterday in which he used the term ‘cognitive poverty’ to define those at the very bottom of the learning distribution, below a conceptually equivalent (not yet precisely measured) ‘cognitive poverty line’. Using PISA data, he documents that the big difference between the worst countries on PISA and middling countries is precisely at the bottom of the distribution - countries with better average scores don’t have high levels of very low learning (level 1 and 2 on the PISA scale), but don’t do that much better at the highest levels.

But when people try and design solutions that might help a whole bunch of people get just across that poverty line, say from level 1 or 2 to level 3 or 4 (like, say, scripted lessons), there is dramatic push-back from many in education. Basic skills aren’t enough, we can’t just define low-bar learning goals, we need to develop children holistically with creative problem solving 21st century skills and art lessons, and all children should be taught by Robin Williams from Dead Poet’s Society.

Why have global poverty advocates been so successful at re-orientating an industry, but cognitive poverty advocates so unsuccessful?

A research agenda on education & institutions

From Tessa Bold & Jakob Svensson for the DFID-OPM-Paris School of Economics research programme "EDI"

 

1. A focus on learning in primary is still essential - don’t get too distracted by secondary and tertiary

2. More focus on teachers’ effort, knowledge, and skills

3. How do we go from pilots to scaled-up programs? (and related - can we design interventions that explicitly allow for existing implementation constraints at scale)

4. How can we use ICT to bring down the cost of sharing information on performance?

5. More research on public-private partnerships such as voucher programs

The key to better education systems is accountability. So how on earth do we do that?

And what do we even actually mean when we talk about accountability?

Perhaps the key theme emerging from research on reforming education systems is accountability. But accountability means different things to different people. To start with, many think first of bottom-up (‘citizen’ or ‘social’) accountability. But increasingly in development economics, enthusiasm is waning for bottom-up social accountability as studies show limited impacts on outcomes. The implicit conclusion then is to revisit top-down (state) accountability. As Rachel Glennerster (Executive Director of J-PAL) wrote recently: 

"For years the Bank and other international agencies have sought to give the poor a voice in health, education, and infrastructure decisions through channels unrelated to politics. They have set up school committees, clinic committees, water and sanitation committees on which sit members of the local community. These members are then asked to “oversee” the work of teachers, health workers, and others. But a body of research suggests that this approach has produced disappointing results."

One striking example of this kind of research is Ben Olken’s work on infrastructure in Indonesia, which directly compared the effect of a top-down audit (which was effective) with bottom-up community monitoring (ineffective).

So what do we mean by top-down accountability for schools?

Within top-down accountability there are a range of methods by which schools and teachers could be held accountable for their performance. Three broad types stand out:

• Student test scores (whether simple averages or more sophisticated value-added models)
• Professional judgement (e.g. based on lesson observations)
• Student feedback

The Gates Foundation published a major report in 2013 on how to “Measure Effective Teaching”, concluding that each of these three types of measurement has strengths and weaknesses, and that the best teacher evaluation system should therefore combine all three: test scores, lesson observations, and student feedback.

By contrast, when it comes to holding head teachers accountable for school performance, the focus in both US policy reform and research is almost entirely on test scores. There are good reasons for this - education in the US has developed as a fundamentally local activity built on bottom up accountability, often with small and relatively autonomous school districts, with little tradition of supervision by higher levels of government. Nevertheless, as Helen Ladd, a Professor of Public Policy and Economics at Duke University and an expert in school accountability, wrote on the Brookings blog last year:

"The current test based approach to accountability is far too narrow … has led to many unintended and negative consequences. It has narrowed the curriculum, induced schools and teachers to focus on what is being tested, led to teaching to the test, induced schools to manipulate the testing pool, and in some well-publicized cases induced some school teachers and administrators to cheat. 

Now is the time to experiment with inspections for school accountability … 

Such systems have been used extensively in other countries … provide useful information to schools … disseminate information on best practices … draw attention to school activities that have the potential to generate a broader range of educational outcomes than just performance on test scores … [and] treats schools fairly by holding them accountable only for the practices under their control … 

The few studies that have focused on the single narrow measure of student test scores have found small positive effects."

A report by the US think tank “Education Sector” also highlights the value of feedback provided through inspection systems to schools.

"Like many of its American counterparts, Peterhouse Primary School in Norfolk County, England, received some bad news early in 2010. Peterhouse had failed to pass muster under its government’s school accountability scheme, and it would need to take special measures to improve. But that is where the similarity ended. As Peterhouse’s leaders worked to develop an action plan for improving, they benefited from a resource few, if any, American schools enjoy. Bundled right along with the school’s accountability rating came a 14-page narrative report on the school’s specific strengths and weaknesses in key areas, such as leadership and classroom teaching, along with a list of top-priority recommendations for tackling problems. With the report in hand, Peterhouse improved rapidly, taking only 14 months to boost its rating substantially."

In the UK, ‘Ofsted’ reports are based on a composite of several different dimensions, including test scores, but also as importantly, independent assessments of school leadership, teaching practices and support for vulnerable students.

There is a huge lack of evidence on school accountability

This blind spot on school inspections isn’t just a problem for education in the US, though. The US is also home to most of the leading researchers on education in developing countries, and that research agenda is skewed by the US policy and research context. The leading education economists don’t study inspections because there aren’t any in the places they live.

The best literature reviews in economics can often be found in the “Handbook of Economics” series and the Journal of Economic Perspectives (JEP). The Handbook article on "School Accountability" from 2011 exclusively discusses the kind of test-based accountability that is common in the US, with no mention of the kind of inspections common in Europe and other countries at all. A recent JEP symposium on Schools and Accountability includes a great article by Isaac Mbiti, a Research on Improving Systems of Education (RISE) researcher, on ’The Need for Accountability in Education in Developing Countries” which includes; however, only one paragraph on school inspections. Another great resource on this topic is the 2011 World Bank book, "Making Schools Work: New Evidence on Accountability Reforms”. This 'must-read' 250-page book has only two paragraphs on school inspections.

This is in part a disciplinary point - it is mostly a blind-spot of economists. School inspections have been studied in more detail by education researchers. But economists have genuinely raised the bar in terms of using rigorous quantitative methods to study education. In total, I count 7 causal studies of the effects of inspections on learning outcomes - 3 by economists and 4 by education researchers.

Putting aside learning outcomes for a moment, one study from leading RISE researchers, Karthik Muralidharan and Jishnu Das (with Alaka Holla and Aakash Mohpal), in rural India finds that “increases in the frequency of inspections are strongly correlated with lower teacher absence”, which could be expected to lead to more learning as a result. However, no such correlation was found for other countries in a companion study (Bangladesh, Ecuador, Indonesia, Peru, and Uganda).

There is also fascinating qualitative work by fellow RISE researcher, Yamini Aiyar (Director of the ‘Accountability Initiative’ and collaborator of RISE researchers Rukmini Banerji, Karthik Muralidharan, and Lant Pritchett) and co-authors, that looks into how local level education administrators view their role in the Indian state of Bihar. The most frequently used term by local officials to describe their role was a “Post Officer” - someone who simply passes messages up and down the bureaucratic chain - “a powerless cog in a large machine with little authority to take decisions." A survey of their time use found that on average a school visit lasts around one hour, with 15 minutes of that time spent in a classroom, with the rest spent “checking attendance registers, examining the mid-day meal scheme and engaging in casual conversations with headmasters and teacher colleagues … the process of school visits was reduced to a mechanical exercise of ticking boxes and collecting relevant data. Academic 'mentoring' of teachers was not part of the agenda.”

At the Education Partnerships Group (EPG) and RISE we’re hoping to help fill this policy and research gap, through nascent school evaluation reforms supported by EPG in Madhya Pradesh, India, that will be studied by the RISE India research team, and an ongoing reform project working with the government of the Western Cape in South Africa. Everything we know about education systems in developing countries suggests that they are in crisis, and that a key part of the solution is around accountability. Yet we know little about how school inspections - the main component of school accountability in most developed countries - might be more effective in poor countries. It’s time we changed that.

This post appeared first on the RISE website. 

Why School Systems Matter, and How We Can Fix Them

This post first appeared on the CGD blog Views from the Center

Accountability in school systems is essential to deliver better learning and accelerate progress in developing countries. What is still really lacking—and what RISE is working towards (such as with Lant Pritchett’s coherence paper)—is a coherent and complete analytical framework capturing the key elements of a system of school accountability that can explain the divergent experiences we have seen in school reform.

The latest issue of the Journal of Economic Perspectives has a great symposium on schools and accountability, covering much of the research that motivated the development of RISE. Here are a few of the research highlights:

• Isaac Mbiti (part of the RISE Tanzania team) discusses “The Need for Accountability in Education in Developing Countries.”
   
• Brian Jacob & Jesse Rothstein dig into how we measure student ability in modern assessment systems (including some helpful discussion of what IRT can and can’t do).
   
• David Deming & David Figlio draw lessons from US experience with test-based school accountability systems, including caution about unintended consequences and ‘gaming,’ and noting that accountability seems to work best for low-performers.
   
• Julia Chabrier, Sarah Cohodes, and Philip Oreopoulos sum up what we can learn from charter school lotteries, re-emphasizing the point that charter schools seem to work best in neighbourhoods with poor performing schools.

Here though I’m going to focus on Ludger Woessmann’s article, "The Importance of School Systems” to help break down the possible reasons for the differences in performance across systems.

1. Why school systems matter: Your country matters for your test score


Woessmann starts by documenting the size of the gap in learning performance of 15-year-olds across countries, using combined PISA and TIMMS scores. As a rule of thumb, children learn on average between 25-30 points per year on this scale, which means that the average 15-year-old in the best performing countries (Singapore & Hong Kong) is roughly two years ahead of the average OECD country (for comparison, the UK scores roughly around this OECD mean).

Down at the other end, the average 15-year-old in Peru or Indonesia is well over 100 points below average—at least four years behind the UK. Ghana and South Africa are more like 200 points below average—6-7 years behind the UK/OECD average. This means that if a typical UK 15-year-old is in year (grade) 10, then the average Ghanaian 15-year old is just at grade 3 level.

Whilst the data for the poor countries we do have is really really bad, for most poor countries we don’t even have any data. In his blog post, “The Case for Global Standardized Testing,” Justin Sandefur highlights just what a tiny percentage of the population of low-income countries are covered by any internationally comparable standardized assessment. It’s essentially indistinguishable from the zero percent of students included in international assessments. Even the regional assessments (which provide some comparability) only cover 40 percent of kids from low-income countries, and only at primary school level. By contrast, 80 percent of children from high-income countries are covered at both primary and secondary level in international tests.

 

2. Country or school factors?

Is the difference in test scores among countries really about school quality though, or rather is it just national wealth, culture, or something else? Woessmann presents a mammoth international “education production function” exercise looking at correlates of test scores across over 200,000 students in 29 countries.
He finds that specific features of systems (such as school autonomy or centralised leaving exams) do correlate with student performance. Further, together these school system variables seem to matter more than student’s family background or school resources.
Moreover, features of the education system seem to matter 3-5 times more as additional variables in explaining outcomes than the level of school spending (adding 0.259 vs. 0.045 in explanatory power).

He also cites Abhijeet Singh’s research (part of the RISE India team), who demonstrates that gaps between students across countries are small when they enter school, and grows as they progress through the school system.

3. Correlation or causation?

The “education production function” approach that Woessmann takes here is explicitly based on correlations as it is difficult to impossible to nail causality down at the aggregate level.
But one first step is to look at changes in variables rather than levels. Take spending: whereas you might be able to find a positive correlation between GDP or the level of education spending and student learning, when you look at increases in spending, there is no correlation with increases in learning—which suggests that just looking at levels rather than changes is confounded by something else (e.g., while richer countries might spend more and have better test scores,  it’s not the spending that causes the test scores,  but insteada third factor correlated with both).

 

4. Money aside, what about other “inputs”?

Spending aside, Woessmann goes on to review what the rest of the literature says about three key inputs (for a look at other inputs, see the RISE working paper by Paul Glewwe & Karthik Muralidharan).

• Class Size: has “a limited role at best"—for example, a 2002 paper by Woessmann used quasi-random variation in small sizes to demonstrate the generally small and heterogeneous effect of class size in different countries.
   
• Instruction Time: is better, and "can increase educational opportunities.”
   
• Teacher Quality: (as measured by gains in student performance) "is related to better student achievement." (It’s worth noting that none of the “thin input” measures of teacher quality (such as qualifications or experience or pay) have substantial effects on learning.)

5. So what features of *systems* leads to more of these inputs and better performance?

• External exams: "A large literature has shown consistent positive associations between external exams and student achievement” (including cross-country analysis but also a cross-subject diff-in-diff approach within Germany).
   
• School autonomy: "School autonomy has a significant effect on student achievement, but this effect varies systematically … part of the negative effect of school autonomy stems from a lack of accountability” (including with a country fixed effects approach looking at changes over time).
   
• Private competition: "Cross-country evidence suggests a strong association of achievement levels with the share of privately operated schools” (robust to exogenous variation based on historical differences in religion).
   
• Tracking: "earlier tracking tends to raise the inequality of educational outcomes."

All of these points reiterate those made in various RISE-framing documents. School systems really matter—the differences in performance across systems are huge, and much of the gap seems to be attributable to real differences in school quality across systems, rather than other country-specific characteristics. And much of the discrepancy in school quality seems to be related to aspects of systems around accountability.
RISE is continuing to sort out and analyze these diverging experiences in school reform—and more importantly how developing countries can accelerate their progress towards global standards of learning.

What can we learn from Sociology about learning?

My summary on the RISE blog of the working paper from the RISE team main sociologist Susan Watkins (& Amy Kaler):

"One of the key things that should strike you from this particular framework and this way of disaggregating the WDR04 accountability triangle, is that the Delegation aspect of accountability relationships in education systems is probably the least studied. The WDR04 has spawned a range of literature looking at the role of better information and better incentives for performance, and there has been plenty of research looking at financing and resourcing. But there’s next to nothing on delegation - what do parents and governments actually expect from schools? Much of the research on the economics of education looks at the effects of schooling on later outcomes, for example on earnings or health, but that is not the same question as what was initially intended. 

This is where the RISE working paper by Susan Watkins and Amy Kaler comes in. Their paper focuses on the collective understandings of education by teachers, parents, and students."

Do Books “Work”?

It might seem obvious to some of you reading this that it might be possible to learn something from a book. But as a recent review for RISE by Paul Glewwe and Karthik Muralidharan found, researchers have actually so far failed to show rigorously that there is any improvement in test scores in developing countries after handing out textbooks to schools. There have now been four different Randomized Controlled Trials showing no improvement (and for four different reasons).

So when might books “work”? A new paper from the World Bank proclaims to answer just this question: “When Do In-service Teacher Training and Books Improve Student Achievement? Experimental Evidence from Mongolia.”

Their answer, somewhat disappointingly, seems to be “when it happens in Mongolia”. More constructively, though books alone seem to work, they work better when combined with teacher training, reinforcing Glewwe and Muralidharan’s conclusion about the importance of complementarities.

But that “there are complementarities” isn’t a very satisfying conclusion by itself. The more comprehensive hypothesis being developed for RISE by Lant Pritchett is: “when there is an accountability framework which is coherent for learning” - that is, when all of the relevant actors are held accountable for common goals through clear delegation of those goals, and have the resources to accomplish them. We’re hoping that this accountability coherency system diagnostic can be a useful tool for thinking through systematically what it is about specific contexts that mean that interventions work in some places and not others. What is it about Mongolia which means that providing books alone can be enough, in contrast to those other studies in Kenya and Sierra Leone? It just might be a coherent accountability system.

This post was first published on the RISE blog

First RISE Working Papers

The first set of working papers from RISE (Research on Improving Systems of Education) are out.

Paul Glewwe and Karthik Muralidharan have an updated review of everything we know about rigorous evaluations of interventions to improve learning in developing countries (paper here, my comment on the RISE blog here).

Rukmini Banerji describes how a disruptive pedagogical innovation spread (and didn’t) in Bihar, and Kara Hanson tell us what education can learn from health systems research.

Mari Oye also has a blogpost up about the UN Myworld survey and the SDGs.

Coming soon, Lant’s consolidated explanation of what an Education System actually is, grand general theory of why some things work sometimes but not all the time, and tentative framework for diagnosing systems for constraints and prioritising action. Watch this space.