The key to better education systems is accountability. So how on earth do we do that?

And what do we even actually mean when we talk about accountability?

Perhaps the key theme emerging from research on reforming education systems is accountability. But accountability means different things to different people. To start with, many think first of bottom-up (‘citizen’ or ‘social’) accountability. But increasingly in development economics, enthusiasm is waning for bottom-up social accountability as studies show limited impacts on outcomes. The implicit conclusion then is to revisit top-down (state) accountability. As Rachel Glennerster (Executive Director of J-PAL) wrote recently: 

"For years the Bank and other international agencies have sought to give the poor a voice in health, education, and infrastructure decisions through channels unrelated to politics. They have set up school committees, clinic committees, water and sanitation committees on which sit members of the local community. These members are then asked to “oversee” the work of teachers, health workers, and others. But a body of research suggests that this approach has produced disappointing results."

One striking example of this kind of research is Ben Olken’s work on infrastructure in Indonesia, which directly compared the effect of a top-down audit (which was effective) with bottom-up community monitoring (ineffective).

So what do we mean by top-down accountability for schools?

Within top-down accountability there are a range of methods by which schools and teachers could be held accountable for their performance. Three broad types stand out:

• Student test scores (whether simple averages or more sophisticated value-added models)
• Professional judgement (e.g. based on lesson observations)
• Student feedback

The Gates Foundation published a major report in 2013 on how to “Measure Effective Teaching”, concluding that each of these three types of measurement has strengths and weaknesses, and that the best teacher evaluation system should therefore combine all three: test scores, lesson observations, and student feedback.

By contrast, when it comes to holding head teachers accountable for school performance, the focus in both US policy reform and research is almost entirely on test scores. There are good reasons for this - education in the US has developed as a fundamentally local activity built on bottom up accountability, often with small and relatively autonomous school districts, with little tradition of supervision by higher levels of government. Nevertheless, as Helen Ladd, a Professor of Public Policy and Economics at Duke University and an expert in school accountability, wrote on the Brookings blog last year:

"The current test based approach to accountability is far too narrow … has led to many unintended and negative consequences. It has narrowed the curriculum, induced schools and teachers to focus on what is being tested, led to teaching to the test, induced schools to manipulate the testing pool, and in some well-publicized cases induced some school teachers and administrators to cheat. 

Now is the time to experiment with inspections for school accountability … 

Such systems have been used extensively in other countries … provide useful information to schools … disseminate information on best practices … draw attention to school activities that have the potential to generate a broader range of educational outcomes than just performance on test scores … [and] treats schools fairly by holding them accountable only for the practices under their control … 

The few studies that have focused on the single narrow measure of student test scores have found small positive effects."

A report by the US think tank “Education Sector” also highlights the value of feedback provided through inspection systems to schools.

"Like many of its American counterparts, Peterhouse Primary School in Norfolk County, England, received some bad news early in 2010. Peterhouse had failed to pass muster under its government’s school accountability scheme, and it would need to take special measures to improve. But that is where the similarity ended. As Peterhouse’s leaders worked to develop an action plan for improving, they benefited from a resource few, if any, American schools enjoy. Bundled right along with the school’s accountability rating came a 14-page narrative report on the school’s specific strengths and weaknesses in key areas, such as leadership and classroom teaching, along with a list of top-priority recommendations for tackling problems. With the report in hand, Peterhouse improved rapidly, taking only 14 months to boost its rating substantially."

In the UK, ‘Ofsted’ reports are based on a composite of several different dimensions, including test scores, but also as importantly, independent assessments of school leadership, teaching practices and support for vulnerable students.

There is a huge lack of evidence on school accountability

This blind spot on school inspections isn’t just a problem for education in the US, though. The US is also home to most of the leading researchers on education in developing countries, and that research agenda is skewed by the US policy and research context. The leading education economists don’t study inspections because there aren’t any in the places they live.

The best literature reviews in economics can often be found in the “Handbook of Economics” series and the Journal of Economic Perspectives (JEP). The Handbook article on "School Accountability" from 2011 exclusively discusses the kind of test-based accountability that is common in the US, with no mention of the kind of inspections common in Europe and other countries at all. A recent JEP symposium on Schools and Accountability includes a great article by Isaac Mbiti, a Research on Improving Systems of Education (RISE) researcher, on ’The Need for Accountability in Education in Developing Countries” which includes; however, only one paragraph on school inspections. Another great resource on this topic is the 2011 World Bank book, "Making Schools Work: New Evidence on Accountability Reforms”. This 'must-read' 250-page book has only two paragraphs on school inspections.

This is in part a disciplinary point - it is mostly a blind-spot of economists. School inspections have been studied in more detail by education researchers. But economists have genuinely raised the bar in terms of using rigorous quantitative methods to study education. In total, I count 7 causal studies of the effects of inspections on learning outcomes - 3 by economists and 4 by education researchers.

Putting aside learning outcomes for a moment, one study from leading RISE researchers, Karthik Muralidharan and Jishnu Das (with Alaka Holla and Aakash Mohpal), in rural India finds that “increases in the frequency of inspections are strongly correlated with lower teacher absence”, which could be expected to lead to more learning as a result. However, no such correlation was found for other countries in a companion study (Bangladesh, Ecuador, Indonesia, Peru, and Uganda).

There is also fascinating qualitative work by fellow RISE researcher, Yamini Aiyar (Director of the ‘Accountability Initiative’ and collaborator of RISE researchers Rukmini Banerji, Karthik Muralidharan, and Lant Pritchett) and co-authors, that looks into how local level education administrators view their role in the Indian state of Bihar. The most frequently used term by local officials to describe their role was a “Post Officer” - someone who simply passes messages up and down the bureaucratic chain - “a powerless cog in a large machine with little authority to take decisions." A survey of their time use found that on average a school visit lasts around one hour, with 15 minutes of that time spent in a classroom, with the rest spent “checking attendance registers, examining the mid-day meal scheme and engaging in casual conversations with headmasters and teacher colleagues … the process of school visits was reduced to a mechanical exercise of ticking boxes and collecting relevant data. Academic 'mentoring' of teachers was not part of the agenda.”

At the Education Partnerships Group (EPG) and RISE we’re hoping to help fill this policy and research gap, through nascent school evaluation reforms supported by EPG in Madhya Pradesh, India, that will be studied by the RISE India research team, and an ongoing reform project working with the government of the Western Cape in South Africa. Everything we know about education systems in developing countries suggests that they are in crisis, and that a key part of the solution is around accountability. Yet we know little about how school inspections - the main component of school accountability in most developed countries - might be more effective in poor countries. It’s time we changed that.

This post appeared first on the RISE website. 

The reductive seduction of OPP (Other people’s problems)

Courtney Martin has an interesting post at the Development Set about the “reductive seduction” of other people’s problems. Problems we know something about (gun control in America as her example for the Americans) seem complex, political, and intractable, whereas problems we know less about (rural hunger or girl’s secondary education or homophobia in Uganda) seem more straightforward.

Which reminds me a little of the difference between statements from leaders on education in their own country and in other people’s countries.

Here’s Julia Gillard on education in Australia:

"We need a commitment to transparency and accountability. It's my strong view that lack of transparency both hides failure and helps us ignore it. It feeds a culture where all the adults involved - the teachers, the principals, the community leaders and the members of parliament - avoid accountability. And lack of transparency prevents us from identifying where greater effort and investment are needed. Importantly, transparency and accountability are overwhelmingly supported by parents."

And Julia Gillard on education in developing countries:

"More money.

Constantly improving ways of working.

An even greater embrace of the power of partnership."

Gordon Brown on education in the UK

"we will focus on classroom standards, ensure that we monitor exam standards rigorously, reform the qualifications system … World class education, we know, achieves high standards for 100% of the children when there are systems of accountability, funding and pupil tracking that leave no child behind and personalised learning is tailored to the unique potential of every child with one-to-one tutoring and support. That world class education depends on a systematic intolerance of failure and a preparedness of public authorities to intervene and to innovate to eradicate failure"

And Gordon Brown on education in developing countries

"The biggest obstacle to what would be a spectacular achievement - as is so often the case - is a shortage of money."

Do Books “Work”?

It might seem obvious to some of you reading this that it might be possible to learn something from a book. But as a recent review for RISE by Paul Glewwe and Karthik Muralidharan found, researchers have actually so far failed to show rigorously that there is any improvement in test scores in developing countries after handing out textbooks to schools. There have now been four different Randomized Controlled Trials showing no improvement (and for four different reasons).

So when might books “work”? A new paper from the World Bank proclaims to answer just this question: “When Do In-service Teacher Training and Books Improve Student Achievement? Experimental Evidence from Mongolia.”

Their answer, somewhat disappointingly, seems to be “when it happens in Mongolia”. More constructively, though books alone seem to work, they work better when combined with teacher training, reinforcing Glewwe and Muralidharan’s conclusion about the importance of complementarities.

But that “there are complementarities” isn’t a very satisfying conclusion by itself. The more comprehensive hypothesis being developed for RISE by Lant Pritchett is: “when there is an accountability framework which is coherent for learning” - that is, when all of the relevant actors are held accountable for common goals through clear delegation of those goals, and have the resources to accomplish them. We’re hoping that this accountability coherency system diagnostic can be a useful tool for thinking through systematically what it is about specific contexts that mean that interventions work in some places and not others. What is it about Mongolia which means that providing books alone can be enough, in contrast to those other studies in Kenya and Sierra Leone? It just might be a coherent accountability system.

This post was first published on the RISE blog

Innovative education financing modalities from 1862

Apparently payment by results isn’t quite so new.

"The reference here is to England’s Payment by Results school reform of 1862. According to the Revised code of the Department of Education in Britain in 1862, capitation grants to schools were reduced and payments were made to school on the basis of students passing on-site examinations given by inspectors in reading, writing and arithmetic. There has been much debate among historians about what the payment for results reform really accomplished. Mitch (2010) looks at educational performance across British counties over the 30 years of the policy and shows that during this time, inequalities across counties declined. But in the absence of data on trends prior to the reform, it is hard to establish whether this was a consequence of the reforms. In contrast, the quote here paraphrases Matthew Arnold, a poet and school inspector who returns from a trip to France and notes: “I find in English schools…..a deadness, a slackness and a discouragement….This change is certainly to be attributed to the `Payment by Results’ school legislation of 1862.”(Great Britain Privy Council 1868, Page 290)."

From a new paper by Andrabi, Das, & Khwaja