Why is there no interest in kinky learning?

Just *how* poor are *your* beneficiaries though? In the aid project business everybody is obsessed with reaching the *poorest* of the poor. The ultra poor. The extreme poor. Lant Pritchett has criticised extensively this arbitrary focus on getting people above a certain threshold, as if the people earning $1.91 a day (just above the international poverty line) really have substantively better lives than those on $1.89 (just below). Instead he argues we should be focusing on economic growth and lifting the whole distribution, with perhaps a much higher global poverty line to aim at of around $10-15 a day, roughly the poverty line in rich countries.

Weirdly, we have the opposite problem in global education, where it is impossible to get people to focus on small incremental gains for those at the bottom of the learning distribution. Luis Crouch gave a great talk at a RISE event in Oxford yesterday in which he used the term ‘cognitive poverty’ to define those at the very bottom of the learning distribution, below a conceptually equivalent (not yet precisely measured) ‘cognitive poverty line’. Using PISA data, he documents that the big difference between the worst countries on PISA and middling countries is precisely at the bottom of the distribution - countries with better average scores don’t have high levels of very low learning (level 1 and 2 on the PISA scale), but don’t do that much better at the highest levels.

But when people try and design solutions that might help a whole bunch of people get just across that poverty line, say from level 1 or 2 to level 3 or 4 (like, say, scripted lessons), there is dramatic push-back from many in education. Basic skills aren’t enough, we can’t just define low-bar learning goals, we need to develop children holistically with creative problem solving 21st century skills and art lessons, and all children should be taught by Robin Williams from Dead Poet’s Society.

Why have global poverty advocates been so successful at re-orientating an industry, but cognitive poverty advocates so unsuccessful?

The key to better education systems is accountability. So how on earth do we do that?

And what do we even actually mean when we talk about accountability?

Perhaps the key theme emerging from research on reforming education systems is accountability. But accountability means different things to different people. To start with, many think first of bottom-up (‘citizen’ or ‘social’) accountability. But increasingly in development economics, enthusiasm is waning for bottom-up social accountability as studies show limited impacts on outcomes. The implicit conclusion then is to revisit top-down (state) accountability. As Rachel Glennerster (Executive Director of J-PAL) wrote recently: 

"For years the Bank and other international agencies have sought to give the poor a voice in health, education, and infrastructure decisions through channels unrelated to politics. They have set up school committees, clinic committees, water and sanitation committees on which sit members of the local community. These members are then asked to “oversee” the work of teachers, health workers, and others. But a body of research suggests that this approach has produced disappointing results."

One striking example of this kind of research is Ben Olken’s work on infrastructure in Indonesia, which directly compared the effect of a top-down audit (which was effective) with bottom-up community monitoring (ineffective).

So what do we mean by top-down accountability for schools?

Within top-down accountability there are a range of methods by which schools and teachers could be held accountable for their performance. Three broad types stand out:

• Student test scores (whether simple averages or more sophisticated value-added models)
• Professional judgement (e.g. based on lesson observations)
• Student feedback

The Gates Foundation published a major report in 2013 on how to “Measure Effective Teaching”, concluding that each of these three types of measurement has strengths and weaknesses, and that the best teacher evaluation system should therefore combine all three: test scores, lesson observations, and student feedback.

By contrast, when it comes to holding head teachers accountable for school performance, the focus in both US policy reform and research is almost entirely on test scores. There are good reasons for this - education in the US has developed as a fundamentally local activity built on bottom up accountability, often with small and relatively autonomous school districts, with little tradition of supervision by higher levels of government. Nevertheless, as Helen Ladd, a Professor of Public Policy and Economics at Duke University and an expert in school accountability, wrote on the Brookings blog last year:

"The current test based approach to accountability is far too narrow … has led to many unintended and negative consequences. It has narrowed the curriculum, induced schools and teachers to focus on what is being tested, led to teaching to the test, induced schools to manipulate the testing pool, and in some well-publicized cases induced some school teachers and administrators to cheat. 

Now is the time to experiment with inspections for school accountability … 

Such systems have been used extensively in other countries … provide useful information to schools … disseminate information on best practices … draw attention to school activities that have the potential to generate a broader range of educational outcomes than just performance on test scores … [and] treats schools fairly by holding them accountable only for the practices under their control … 

The few studies that have focused on the single narrow measure of student test scores have found small positive effects."

A report by the US think tank “Education Sector” also highlights the value of feedback provided through inspection systems to schools.

"Like many of its American counterparts, Peterhouse Primary School in Norfolk County, England, received some bad news early in 2010. Peterhouse had failed to pass muster under its government’s school accountability scheme, and it would need to take special measures to improve. But that is where the similarity ended. As Peterhouse’s leaders worked to develop an action plan for improving, they benefited from a resource few, if any, American schools enjoy. Bundled right along with the school’s accountability rating came a 14-page narrative report on the school’s specific strengths and weaknesses in key areas, such as leadership and classroom teaching, along with a list of top-priority recommendations for tackling problems. With the report in hand, Peterhouse improved rapidly, taking only 14 months to boost its rating substantially."

In the UK, ‘Ofsted’ reports are based on a composite of several different dimensions, including test scores, but also as importantly, independent assessments of school leadership, teaching practices and support for vulnerable students.

There is a huge lack of evidence on school accountability

This blind spot on school inspections isn’t just a problem for education in the US, though. The US is also home to most of the leading researchers on education in developing countries, and that research agenda is skewed by the US policy and research context. The leading education economists don’t study inspections because there aren’t any in the places they live.

The best literature reviews in economics can often be found in the “Handbook of Economics” series and the Journal of Economic Perspectives (JEP). The Handbook article on "School Accountability" from 2011 exclusively discusses the kind of test-based accountability that is common in the US, with no mention of the kind of inspections common in Europe and other countries at all. A recent JEP symposium on Schools and Accountability includes a great article by Isaac Mbiti, a Research on Improving Systems of Education (RISE) researcher, on ’The Need for Accountability in Education in Developing Countries” which includes; however, only one paragraph on school inspections. Another great resource on this topic is the 2011 World Bank book, "Making Schools Work: New Evidence on Accountability Reforms”. This 'must-read' 250-page book has only two paragraphs on school inspections.

This is in part a disciplinary point - it is mostly a blind-spot of economists. School inspections have been studied in more detail by education researchers. But economists have genuinely raised the bar in terms of using rigorous quantitative methods to study education. In total, I count 7 causal studies of the effects of inspections on learning outcomes - 3 by economists and 4 by education researchers.

Putting aside learning outcomes for a moment, one study from leading RISE researchers, Karthik Muralidharan and Jishnu Das (with Alaka Holla and Aakash Mohpal), in rural India finds that “increases in the frequency of inspections are strongly correlated with lower teacher absence”, which could be expected to lead to more learning as a result. However, no such correlation was found for other countries in a companion study (Bangladesh, Ecuador, Indonesia, Peru, and Uganda).

There is also fascinating qualitative work by fellow RISE researcher, Yamini Aiyar (Director of the ‘Accountability Initiative’ and collaborator of RISE researchers Rukmini Banerji, Karthik Muralidharan, and Lant Pritchett) and co-authors, that looks into how local level education administrators view their role in the Indian state of Bihar. The most frequently used term by local officials to describe their role was a “Post Officer” - someone who simply passes messages up and down the bureaucratic chain - “a powerless cog in a large machine with little authority to take decisions." A survey of their time use found that on average a school visit lasts around one hour, with 15 minutes of that time spent in a classroom, with the rest spent “checking attendance registers, examining the mid-day meal scheme and engaging in casual conversations with headmasters and teacher colleagues … the process of school visits was reduced to a mechanical exercise of ticking boxes and collecting relevant data. Academic 'mentoring' of teachers was not part of the agenda.”

At the Education Partnerships Group (EPG) and RISE we’re hoping to help fill this policy and research gap, through nascent school evaluation reforms supported by EPG in Madhya Pradesh, India, that will be studied by the RISE India research team, and an ongoing reform project working with the government of the Western Cape in South Africa. Everything we know about education systems in developing countries suggests that they are in crisis, and that a key part of the solution is around accountability. Yet we know little about how school inspections - the main component of school accountability in most developed countries - might be more effective in poor countries. It’s time we changed that.

This post appeared first on the RISE website. 

Why School Systems Matter, and How We Can Fix Them

This post first appeared on the CGD blog Views from the Center

Accountability in school systems is essential to deliver better learning and accelerate progress in developing countries. What is still really lacking—and what RISE is working towards (such as with Lant Pritchett’s coherence paper)—is a coherent and complete analytical framework capturing the key elements of a system of school accountability that can explain the divergent experiences we have seen in school reform.

The latest issue of the Journal of Economic Perspectives has a great symposium on schools and accountability, covering much of the research that motivated the development of RISE. Here are a few of the research highlights:

• Isaac Mbiti (part of the RISE Tanzania team) discusses “The Need for Accountability in Education in Developing Countries.”
   
• Brian Jacob & Jesse Rothstein dig into how we measure student ability in modern assessment systems (including some helpful discussion of what IRT can and can’t do).
   
• David Deming & David Figlio draw lessons from US experience with test-based school accountability systems, including caution about unintended consequences and ‘gaming,’ and noting that accountability seems to work best for low-performers.
   
• Julia Chabrier, Sarah Cohodes, and Philip Oreopoulos sum up what we can learn from charter school lotteries, re-emphasizing the point that charter schools seem to work best in neighbourhoods with poor performing schools.

Here though I’m going to focus on Ludger Woessmann’s article, "The Importance of School Systems” to help break down the possible reasons for the differences in performance across systems.

1. Why school systems matter: Your country matters for your test score


Woessmann starts by documenting the size of the gap in learning performance of 15-year-olds across countries, using combined PISA and TIMMS scores. As a rule of thumb, children learn on average between 25-30 points per year on this scale, which means that the average 15-year-old in the best performing countries (Singapore & Hong Kong) is roughly two years ahead of the average OECD country (for comparison, the UK scores roughly around this OECD mean).

Down at the other end, the average 15-year-old in Peru or Indonesia is well over 100 points below average—at least four years behind the UK. Ghana and South Africa are more like 200 points below average—6-7 years behind the UK/OECD average. This means that if a typical UK 15-year-old is in year (grade) 10, then the average Ghanaian 15-year old is just at grade 3 level.

Whilst the data for the poor countries we do have is really really bad, for most poor countries we don’t even have any data. In his blog post, “The Case for Global Standardized Testing,” Justin Sandefur highlights just what a tiny percentage of the population of low-income countries are covered by any internationally comparable standardized assessment. It’s essentially indistinguishable from the zero percent of students included in international assessments. Even the regional assessments (which provide some comparability) only cover 40 percent of kids from low-income countries, and only at primary school level. By contrast, 80 percent of children from high-income countries are covered at both primary and secondary level in international tests.

 

2. Country or school factors?

Is the difference in test scores among countries really about school quality though, or rather is it just national wealth, culture, or something else? Woessmann presents a mammoth international “education production function” exercise looking at correlates of test scores across over 200,000 students in 29 countries.
He finds that specific features of systems (such as school autonomy or centralised leaving exams) do correlate with student performance. Further, together these school system variables seem to matter more than student’s family background or school resources.
Moreover, features of the education system seem to matter 3-5 times more as additional variables in explaining outcomes than the level of school spending (adding 0.259 vs. 0.045 in explanatory power).

He also cites Abhijeet Singh’s research (part of the RISE India team), who demonstrates that gaps between students across countries are small when they enter school, and grows as they progress through the school system.

3. Correlation or causation?

The “education production function” approach that Woessmann takes here is explicitly based on correlations as it is difficult to impossible to nail causality down at the aggregate level.
But one first step is to look at changes in variables rather than levels. Take spending: whereas you might be able to find a positive correlation between GDP or the level of education spending and student learning, when you look at increases in spending, there is no correlation with increases in learning—which suggests that just looking at levels rather than changes is confounded by something else (e.g., while richer countries might spend more and have better test scores,  it’s not the spending that causes the test scores,  but insteada third factor correlated with both).

 

4. Money aside, what about other “inputs”?

Spending aside, Woessmann goes on to review what the rest of the literature says about three key inputs (for a look at other inputs, see the RISE working paper by Paul Glewwe & Karthik Muralidharan).

• Class Size: has “a limited role at best"—for example, a 2002 paper by Woessmann used quasi-random variation in small sizes to demonstrate the generally small and heterogeneous effect of class size in different countries.
   
• Instruction Time: is better, and "can increase educational opportunities.”
   
• Teacher Quality: (as measured by gains in student performance) "is related to better student achievement." (It’s worth noting that none of the “thin input” measures of teacher quality (such as qualifications or experience or pay) have substantial effects on learning.)

5. So what features of *systems* leads to more of these inputs and better performance?

• External exams: "A large literature has shown consistent positive associations between external exams and student achievement” (including cross-country analysis but also a cross-subject diff-in-diff approach within Germany).
   
• School autonomy: "School autonomy has a significant effect on student achievement, but this effect varies systematically … part of the negative effect of school autonomy stems from a lack of accountability” (including with a country fixed effects approach looking at changes over time).
   
• Private competition: "Cross-country evidence suggests a strong association of achievement levels with the share of privately operated schools” (robust to exogenous variation based on historical differences in religion).
   
• Tracking: "earlier tracking tends to raise the inequality of educational outcomes."

All of these points reiterate those made in various RISE-framing documents. School systems really matter—the differences in performance across systems are huge, and much of the gap seems to be attributable to real differences in school quality across systems, rather than other country-specific characteristics. And much of the discrepancy in school quality seems to be related to aspects of systems around accountability.
RISE is continuing to sort out and analyze these diverging experiences in school reform—and more importantly how developing countries can accelerate their progress towards global standards of learning.

Reaching universal secondary school won’t solve the learning crisis

Secondary school graduates in Jakarta, urban Ghana, and urban Kenya, have worse literacy skills than primary school graduates in rich countries. More here on the RISE blog.

What can we learn from Sociology about learning?

My summary on the RISE blog of the working paper from the RISE team main sociologist Susan Watkins (& Amy Kaler):

"One of the key things that should strike you from this particular framework and this way of disaggregating the WDR04 accountability triangle, is that the Delegation aspect of accountability relationships in education systems is probably the least studied. The WDR04 has spawned a range of literature looking at the role of better information and better incentives for performance, and there has been plenty of research looking at financing and resourcing. But there’s next to nothing on delegation - what do parents and governments actually expect from schools? Much of the research on the economics of education looks at the effects of schooling on later outcomes, for example on earnings or health, but that is not the same question as what was initially intended. 

This is where the RISE working paper by Susan Watkins and Amy Kaler comes in. Their paper focuses on the collective understandings of education by teachers, parents, and students."

Innovative education financing modalities from 1862

Apparently payment by results isn’t quite so new.

"The reference here is to England’s Payment by Results school reform of 1862. According to the Revised code of the Department of Education in Britain in 1862, capitation grants to schools were reduced and payments were made to school on the basis of students passing on-site examinations given by inspectors in reading, writing and arithmetic. There has been much debate among historians about what the payment for results reform really accomplished. Mitch (2010) looks at educational performance across British counties over the 30 years of the policy and shows that during this time, inequalities across counties declined. But in the absence of data on trends prior to the reform, it is hard to establish whether this was a consequence of the reforms. In contrast, the quote here paraphrases Matthew Arnold, a poet and school inspector who returns from a trip to France and notes: “I find in English schools…..a deadness, a slackness and a discouragement….This change is certainly to be attributed to the `Payment by Results’ school legislation of 1862.”(Great Britain Privy Council 1868, Page 290)."

From a new paper by Andrabi, Das, & Khwaja

The learning crisis in Sierra Leone

"Mohammed’s father is an illiterate petty trader. Although he never got any school himself, he has always been determined that Mohammed [13] should get a good education. When Mohammed joined us, we asked him, as we ask all our students, to complete a word reading assessment. The assessment, which we administer one-to-one in the child’s home, involves reading out a list of 90 words that increase in complexity and difficulty, and from the number and difficulty of the words read correctly an inference can be drawn about the student’s reading age based on UK norms. 

Mohammed got stuck straight away. He barely made it past the first line or two - words like ‘he’, ‘she’, ‘cup’, ‘said’. Mohammed’s father couldn’t read but he could see that his son was struggling and did his best to encourage him. “Try your best Mohammed”, he said. And Mohammed did. He kept trying. But no matter how long he took he couldn’t recognise the words, and eventually we had to call time on the test. Mohammed’s father was heartbroken. “I’ve paid all this money for school,” he said, “but his head is empty.” 

Unfortunately, Mohammed’s story is far from unique."

From an essay by Paul Skidmore, CEO of the Rising Academy Network

Learning goals

Nic Spaull makes the case for one simple learning goal for South Africa:

“Every child must read and write by the end of grade three.”

I think he is absolutely right. You hear often from international education types that we must resist the simplification of goals, and account for broader objectives such as citizenship etc, but the fact remains that the majority of children in Grade 3 in South Africa, and by implication most other developing countries, can't read (and understand) a simple 30 word story such as this one below.  

I actually heard in a meeting at ODI last year that "it would be a tragedy if the post-2015 education goals were reduced to simply all children being able to read and write and do sums." On the contrary I think it would be a tragedy if we let there be any more distraction from ensuring children have the most basic and fundamental skill of being able to read to learn. 

On the political economy of education systems, Nic also posts an interview with a teacher explaining how unions in South Africa control appointments within schools. 

“When you are selecting a Head of Department (HOD) for the school there are 2 parents from the SGB and 1 teacher, the principal is there but cannot vote. In the rural Eastern Cape many of the parents are not well educated. They know nothing about laws so it is just the principal and the teachers. SADTU can very easily influence the parents through the teacher. If SADTU does not get the person that they want they will say there was an irregularity in the interview process. I once encouraged the parents to appoint a good mathematics teacher for my school and they did, but they were not SADTU’s choice so they had the teacher removed. They re-advertised the post but without subject specification because there was no SADTU member who had maths or science. I am now stuck with someone who is babysitting mathematics and my results are terrible. My ANAs are very low in mathematics. And you cannot challenge it.” [“Why can’t you challenge SADTU?”] They will go for you. They will accuse you of sexual misconduct and there must be an enquiry. They will accuse you of financial mismanagement. They will go for small things to catch you. You know you need 3 quotations if you buy something and you must write it down so that if you only have two or forgot to write it down, they will catch you. Most principals will make a small mistake. But these are honest mistakes. But they will catch you.” “The Department is listening and and the union is managing. SADTU does not want to listen, they want to lead and they want to manage.”

How not to improve education in India

Some great analysis from MINT who highlight a new Government of India report, which ranks state education "outcomes".

What is odd is that the government rank has a negative correlation with the rankings of the Pratham report which directly measures learning outcomes.

So what goes into the government "outcomes" index?

- Number of teaching days
- Teacher working hours
- Enrolment rates
- Drop-out rates
- Primary-to-secondary transition rates

These are all basically inputs, with the exception of drop-outs and transition rates, which maybe say something about quality. But none of them are actually directly measuring learning at all. Yet more evidence for the Lant Pritchett case that focusing on inputs or "EMIS-visible" metrics won't get us quality learning outcomes, and measuring learning directly is critical to focusing policy attention on how to improve learning.

HT: Abhijeet Singh

What can be done about India’s failing primary schools?

This is a guest post by Abhijeet Singh, Research Officer at the University of Oxford

It’s that time of the year again. As India’s Annual Survey of Education Report (ASER) was released last week (15th Jan), we are reminded of how abysmal learning levels in the country are. Most children are in school, but very few are learning. Less than half the children in Grade 5 can read a Grade 3 text, and only about a quarter can divide a three-digit number by a one-digit number. And the learning levels seem to be going down every year.

Remedies that won’t work

The real question is what can be done to improve learning outcomes. And the first lesson is that most of what is being proposed is not promising:

Inputs alone won’t improve learning: School reform in India seems to consist mostly of getting kids in school and then improving the inputs they get. But this ‘business-as-usual’ approach doesn’t work. The decline in achievement levels in government schools has coincided with major improvements in inputs. Across developing countries, the link between increasing school resources and student achievement is surprisingly weak: building libraries, providing more money to schools, teaching aids like flipcharts or providing a laptop to every child all don’t seem to improve learning by much, if at all. Upgrading school inputs won’t solve this mess.

Private schooling won’t do too much either: ASER documents, as do a number of studies, that students in private schools do better on average than students in government schools. But these students come from better-off households, with more educated parents - so it is hard to tell whether the schools caused this difference in test scores. 

Two recent studies - one study tracking the same children for 7 years that I recently authored, and another based on a randomized evaluation of school vouchers, both find that private school students do better in a number of dimensions, (English, Science and Hindi) even after background factors are accounted for. But even here the ‘private school effect’ is only a small part of the gap between students’ actual achievement and any objective standards of quality we expect a functioning school system to deliver. And in some key areas, such as Maths, we don’t see any private school effect at all. 

The key constraint is service delivery (but reform is hard!): A quarter of government school teachers seem to not come to school on a given day. And when in school nearly half of the teachers are not teaching. Researchers have looked for solutions: for example, giving teachers performance bonuses, monitoring attendance with cameras, and even giving them detailed information about their students’ learning needs. But even when these interventions seem to have an impact, it is hard to see how these can be scaled up. 

So what might work?

Better pre-school preparation: One new pattern highlighted in ASER this year is that children in government schools seem to start earlier than children in private schools and are less ‘school ready’. A recent paper of mine, published this month in the Oxford Review of Education, finds the same pattern but goes further: gaps between children in the private and government sectors arise before school entry and aren’t just a reflection of household incomes and parental education: at least part of the failure probably comes from the failure of government pre-schools. Investing in early childhood education could be the best investment to make, including in developing countries, but most of India’s anganwadis (pre-school centres) aren’t geared towards education at all.

Supplementary teaching: Karthik Muralidharan, one of the most prominent researchers into primary schooling in India, had a different proposal for India’s Five-Year plan. He suggested supplementing class teachers with more teaching assistants who can provide individual learning support (which has been shown to improve test scores in India). And then to facilitate a system that allows the best of these teaching assistants to become regular teachers and receive preference in hiring. This will not be a cure-all, but it could improve matters significantly. And it could be sustainable not just economically but also politically.

Rethink what is taught and how: A more far-reaching reform, proposed both by the ASER team and by Lant Pritchett, is to rethink what we expect students to learn in primary school. The system focuses on a grade-specific curriculum that requires teachers to complete a list of topics per year, regardless of what students need. The curriculum is targeted at the top 10% of children, the other 90% gain nothing because they are already too far behind. And in a setting where children from many grades sit in the same class, a grade-specific syllabus might not be something to aspire to anyway. Much better to focus on the skills we expect children to acquire, rather than the textbook chapters a teacher can tick off whether or not the children have really understood.

Reforming a situation so dire will not be easy. But the first step, in all of this, is to acknowledge the problem. Perhaps the saddest thing of all about the latest ASER report is the government’s refusal to accept that a problem exists and that it is getting worse. This is not the first time such denials have occurred. India cannot progress without assessing whether children are learning at school and without evaluating what works in the school system and what doesn’t. As reactions to the ASER report demonstrate, India’s researchers and civil society have made that leap. It is time the government does so too.

Where has all the education impact come from?

Lant Pritchett wrote an important paper back in 2000 called "Where has all the education gone." Despite a big increase in schooling in developing countries, there had been little increase on average in productivity.

Now I feel like we have the opposite problem - evidence that going to school has all sorts of wonderful impacts transforming lives

(- saves lives
- improves child nutrition
- increases job opportunities
- reduces child marriage
- reduces early birth
- makes people more tolerant
- leads to economic growth
- leads to more concern with the environment)

alongside evidence that many kids never actually learnt anything at school.

What's the deal? Are the positive impacts driven entirely by the kids who did learn something (the average treatment effect on both learners and non-learners), or is there something special and intangible (non-cognitive skills and character?) which kids can pick up from just sitting in a school even if they don't pick up any reading skills?

Any evidence?

What would Einstein say about schooling for all?

From a letter to his 11-year old son 

I am very pleased that you find joy with the piano. This and carpentry are in my opinion for your age the best pursuits, better even than school. Because those are things which fit a young person such as you very well. Mainly play the things on the piano which please you, even if the teacher does not assign those. That is the way to learn the most, that when you are doing something with such enjoyment that you don’t notice that the time passes.