First and foremost, it should reaffirm its principles: that education should be free; that government should be the guarantor but not necessarily the sole provider of education.
Second, it should focus its education reform efforts on setting high expectations for all children and improving accountability for all schools, whether public or private. School inspections, good data, robust assessments: these are all critical components of a good system accountability and should apply to public and private schools alike.
Third, in contexts in which substantial numbers of children already do attend private schools, the INGO could see if there are any clear market failures in the private sector that could be improved by government or NGO intervention. For example, training of teachers, or addressing information asymmetries through better community engagement.
Fourth, where public-private partnerships are planned or in place, the INGO should challenge government and private sector partners to place equity and quality at the heart of any programme. The UK academy programme started out under the Labour government with a very clear objective: to turn around the 200 worst performing secondary schools in the most deprived areas. This laser sharp focus on closing the achievement gap between rich and poor has changed the conversation about education in the UK: a school cannot be rated as “outstanding” anymore unless it can show it is delivering learning gains for children on free school meals.
Finally, and perhaps most radical of all, the INGO could set up its own network of non-profit PPP schools in a developing country. Demonstrating what high quality education for disadvantaged communities looks like would be a significant value add to any system, particularly if delivered through a PPP with the sustainability and public accountability that should bring. The experience of PEAS schools in Uganda shows that it is possible for an NGO to deliver better management practices and improve performance, at the same cost as local operators and with the inbuilt sustainability that comes from domestic government financing. And Ark’s own experience of running a high performing network of non-profit PPP schools in the UK shows how much an organisation can learn from actually running a set of schools, and how much more effective advisors to governments this can make them. Finally, establishing more non-profit providers in this space might calm some of the heated rhetoric around some of the international for-profit providers.
19 August 2016
The latest issue of the Journal of Economic Perspectives has a great symposium on schools and accountability, covering much of the research that motivated the development of RISE. Here are a few of the research highlights:
- Isaac Mbiti (part of the RISE Tanzania team) discusses “The Need for Accountability in Education in Developing Countries.”
- Brian Jacob & Jesse Rothstein dig into how we measure student ability in modern assessment systems (including some helpful discussion of what IRT can and can’t do).
- David Deming & David Figlio draw lessons from US experience with test-based school accountability systems, including caution about unintended consequences and ‘gaming,’ and noting that accountability seems to work best for low-performers.
- Julia Chabrier, Sarah Cohodes, and Philip Oreopoulos sum up what we can learn from charter school lotteries, re-emphasizing the point that charter schools seem to work best in neighbourhoods with poor performing schools.
Woessmann starts by documenting the size of the gap in learning performance of 15-year-olds across countries, using combined PISA and TIMMS scores. As a rule of thumb, children learn on average between 25-30 points per year on this scale, which means that the average 15-year-old in the best performing countries (Singapore & Hong Kong) is roughly two years ahead of the average OECD country (for comparison, the UK scores roughly around this OECD mean).
Down at the other end, the average 15-year-old in Peru or Indonesia is well over 100 points below average—at least four years behind the UK. Ghana and South Africa are more like 200 points below average—6-7 years behind the UK/OECD average. This means that if a typical UK 15-year-old is in year (grade) 10, then the average Ghanaian 15-year old is just at grade 3 level.
Whilst the data for the poor countries we do have is really really bad, for most poor countries we don’t even have any data. In his blog post, “The Case for Global Standardized Testing,” Justin Sandefur highlights just what a tiny percentage of the population of low-income countries are covered by any internationally comparable standardized assessment. It’s essentially indistinguishable from the zero percent of students included in international assessments. Even the regional assessments (which provide some comparability) only cover 40 percent of kids from low-income countries, and only at primary school level. By contrast, 80 percent of children from high-income countries are covered at both primary and secondary level in international tests.
Is the difference in test scores among countries really about school quality though, or rather is it just national wealth, culture, or something else? Woessmann presents a mammoth international “education production function” exercise looking at correlates of test scores across over 200,000 students in 29 countries.
He finds that specific features of systems (such as school autonomy or centralised leaving exams) do correlate with student performance. Further, together these school system variables seem to matter more than student’s family background or school resources.
Moreover, features of the education system seem to matter 3-5 times more as additional variables in explaining outcomes than the level of school spending (adding 0.259 vs. 0.045 in explanatory power).
He also cites Abhijeet Singh’s research (part of the RISE India team), who demonstrates that gaps between students across countries are small when they enter school, and grows as they progress through the school system.
The “education production function” approach that Woessmann takes here is explicitly based on correlations as it is difficult to impossible to nail causality down at the aggregate level.
But one first step is to look at changes in variables rather than levels. Take spending: whereas you might be able to find a positive correlation between GDP or the level of education spending and student learning, when you look at increases in spending, there is no correlation with increases in learning—which suggests that just looking at levels rather than changes is confounded by something else (e.g., while richer countries might spend more and have better test scores, it’s not the spending that causes the test scores, but insteada third factor correlated with both).
Spending aside, Woessmann goes on to review what the rest of the literature says about three key inputs (for a look at other inputs, see the RISE working paper by Paul Glewwe & Karthik Muralidharan).
- Class Size: has “a limited role at best"—for example, a 2002 paper by Woessmann used quasi-random variation in small sizes to demonstrate the generally small and heterogeneous effect of class size in different countries.
- Instruction Time: is better, and "can increase educational opportunities.”
- Teacher Quality: (as measured by gains in student performance) "is related to better student achievement." (It’s worth noting that none of the “thin input” measures of teacher quality (such as qualifications or experience or pay) have substantial effects on learning.)
- External exams: "A large literature has shown consistent positive associations between external exams and student achievement” (including cross-country analysis but also a cross-subject diff-in-diff approach within Germany).
- School autonomy: "School autonomy has a significant effect on student achievement, but this effect varies systematically … part of the negative effect of school autonomy stems from a lack of accountability” (including with a country fixed effects approach looking at changes over time).
- Private competition: "Cross-country evidence suggests a strong association of achievement levels with the share of privately operated schools” (robust to exogenous variation based on historical differences in religion).
- Tracking: "earlier tracking tends to raise the inequality of educational outcomes."
RISE is continuing to sort out and analyze these diverging experiences in school reform—and more importantly how developing countries can accelerate their progress towards global standards of learning.
16 August 2016
Secondary school graduates in Jakarta, urban Ghana, and urban Kenya, have worse literacy skills than primary school graduates in rich countries. More here on the RISE blog.
08 August 2016
My summary on the RISE blog of the working paper from the RISE team main sociologist Susan Watkins (& Amy Kaler):
"One of the key things that should strike you from this particular framework and this way of disaggregating the WDR04 accountability triangle, is that the Delegation aspect of accountability relationships in education systems is probably the least studied. The WDR04 has spawned a range of literature looking at the role of better information and better incentives for performance, and there has been plenty of research looking at financing and resourcing. But there’s next to nothing on delegation - what do parents and governments actually expect from schools? Much of the research on the economics of education looks at the effects of schooling on later outcomes, for example on earnings or health, but that is not the same question as what was initially intended.
This is where the RISE working paper by Susan Watkins and Amy Kaler comes in. Their paper focuses on the collective understandings of education by teachers, parents, and students."
This is great from Scott Alexander:
"the incorrect model of “foreign cultures being Westernized” casts Western culture as the aggressor, whereas the model of “every culture is being universalized” finds Western culture to be as much a victim as anywhere else. Coca-Cola might have replaced traditional yak’s milk in Mongolia, but it also replaced traditional apple cider in America. A Hopi Indian saddened that her children no longer know the old ritual dances differs little from a Southern Baptist incensed that her kids no longer go to church. Universal values have triumphed over both."
29 July 2016
Thank you for raising this issue with James Duddridge. Unfortunately his reply does not reassure me at all.
The "Better Migration Management" project description notes that "2 reception centres in Gadaref and Kassala, with custody rooms" could "in principle be funded" , and that one of the main risks of the project are that "Provision of equipment and trainings to sensitive national authorities (such as security services or border management) [could be] diverted for repressive aims; [leading to] criticism by NGOs and civil society for engaging with repressive governments on migration (particularly in Eritrea and Sudan)."
Clearly Mr. Duddridge's response, that training will be provided by Western NGOs, and money will not be given directly to the Government of Sudan, does not alleviate this risk of repression. What steps is the government taking to ensure that no irregular migrants will be subjected to repression by the repressive government of Sudan with the use of equipment financed by UK taxpayers?
20 July 2016
This post first appeared on the RISE website
Why do Vietnam school children score over 100 points better on comparable tests than the average for low-income countries?
Vietnam is basically the only low-income country in any of the internationally comparable tests that performs at the same level as rich countries. Vietnam is a massive outlier, performing substantially better than should be expected for a country at that level of income. Rich OECD countries such as the UK and US flock to see the top performing places in the world on the PISA test to try and understand what is so special about education systems in Shanghai and Finland that enables them to perform 100 points better than the OECD average. Vietnam scores over 100 points better than the average for low-income countries.
And this isn’t just on one test – other research by Abhijeet Singh has linked the Oxford Young Lives survey with the international TIMSS test, and again Vietnam massively outperforms the other low-income countries (see chart). Singh’s study shows that the advantage starts early, with Vietnamese children slightly outperforming those in other developing countries before they even start school at age 5, but this gap then grows each year. A year of primary school in Vietnam is considerably more ‘productive’ in terms of skill acquisition than a year of schooling in Peru or India, the paper finds. The question this research raises – and the Vietnam experience suggests - is: “Why is learning-productivity-per-year so much greater in some countries than others?” Or to put it more simply, why are schools so much better in some countries?
A new paper by World Bank researchers Suhas D. Parandekar and Elisabeth K. Sedmik shows just how difficult the “Vietnam effect” is to unscramble. With a statistical decomposition using available measured factors, the research suggests that a combination of targeted investments and “cultural factors” explain roughly half of the huge ~100 point gap between Vietnam and the other low-income countries on the PISA test.
The main factors are:
- Higher level of access to pre-school.
- Investment in school infrastructure, especially in cities and small towns.
And Cultural factors
- Students work harder – skip fewer classes, spend the same or more time in school, plus substantial extra time studying after school. Students are more disciplined and focused on their studies.
- Teachers appear to benefit from closer supervision of their work by the school principal and others.
- Parents may have an important role to play, by taking an active part in combining high expectations of their children, following up with their children’s teachers and contributing at school.
All this only gets us part of the way toward explaining the Vietnam phenomenon. The other half of the answer remains a Vietnam enigma, which we’re hoping the RISE Vietnam research team can help to unravel, ultimately possibly even providing some useful lessons for other low-income countries.
* The chart here shows the average proportion answering a TIMSS question correctly, averaged across 6 questions focused on the number content domain, taken from the 2003 TIMSS assessment
21 June 2016
“Angela Duckworth’s new book Grit: The Power of Passion and Perseverance has been launched with great fanfare, reaching number two on the NY Times Nonfiction bestseller list. She recently gave a very polished and smooth book launch talk to a packed audience at the World Bank, and is working with World Bank colleagues on improving grit in classrooms in Macedonia.”That’s David McKenzie in a great book review, considering what development economists can learn from this hot psychology research trend. Grit – the ability to keep going when things get tough and you aren’t successful straight away – can help explain all sorts of individual outcomes beyond tests of skill or ability. David notes amongst other things how U.S. - centric the research on grit is, and questions how large the effect of grit is even in this context.
So what do we know about the importance of grit in developing countries?
Fortunately, a separate team at the World Bank has recently been rolling out a series of surveys measuring psychological traits including grit alongside measures of skills, income, and other demographics. Data is currently available for 10 countries; Armenia, Bolivia, Colombia, Georgia, Ghana, Kenya, Laos, Sri Lanka, Vietnam, and Yunnan Province (China).
Here’s what I found from some very quick analysis.
Grit and income
I started by looking at the relationship between adults’ grit and their earnings. For seven countries there is no relationship. For the three where some relationship exists, grit explains very little of the variation between the income of individuals. (That is, in the table below, the r2 statistic is less than 0.005.) Adding in a few basic control variables (age, parents’ education and socio-economic status) makes even that weak correlation disappear altogether.
Grit and schooling
Moving next to years of schooling, something more substantial emerges. Grit has a statistically significant relationship with schooling in every country, and throwing in a bunch of control variables doesn’t seem to make it go away. I’m not sure what to make of the magnitude though - less than half a year of extra schooling for a 1-standard-deviation increase in grit. Maybe that’s a lot, maybe it isn't. I suppose the policy-relevant question is how responsive grit might be to interventions.
Grit and learning outcomes
Finally, looking at the correlation with answers on a reading comprehension test. Grit scores pretty poorly here too; one positive correlation, one negative, and eight statistically insignificant. Other control variables by comparison do have the kind of statistically significant relationships you might expect - people tend to score better with more schooling and if they grew up in wealthier families.
I’ll spare you some of the snarkier comments from the office, needless to say that unsurprisingly to some, from a quick look the data does not seem to suggest that grit is all that important in explaining important outcomes in developing countries. Unsurprising, because all the grit and resilience and perseverance in the world is unlikely to help a child succeed at school if they haven’t eaten that day and their teacher hasn’t turned up due to a dysfunctional school system. Similarly in the labour market, individual motivation and grit by itself isn’t going to create any well paying jobs in places where the demand for labour is low because of systemic factors such as bad infrastructure and bad governance.
I will offer one caveat – this measure of grit is based on only three questions rather than Duckworth’s preferred ten, so it is possible that a better measure of grit would matter more. But I doubt it.
And finally – this is not meant as a counsel of despair. For individuals living in low-income countries, of course they should try and persevere as hard as they can to try and achieve their goals. But when it comes to making policy – we should focus on the systemic constraints that are critical to shaping people’s opportunities, rather than just telling them to try harder. Bad schools, infrastructure, and governance, are all fixable public policy problems.
17 May 2016
From a Facebook friend:
So, it turns out that Pakistanis are not eligible to purchase travel insurance online with any insurance company. We are simply not on anyone's list of "eligible countries''. Someone has decided that in addition to not being allowed to enter most of the world, we also do not deserve access to basic financial services. Pakistanis wanting to travel can just f@#k themselves.
Der Spiegel reports… the EU is planning to provide training, equipment, and DETENTION CAMPS to the government of Sudan, which is led by a wanted war criminal, in order that they can prevent human beings from crossing the border out of Sudan in the direction of Europe. The project is to be coordinated by the German development agency GIZ. Wow I feel sorry for the idealists at GIZ who signed up in order to help people, and now they’re building detention facilities for war criminals.
"The ambassadors of the 28 European Union member states had agreed to secrecy. "Under no circumstances" should the public learn what was said at the talks that took place on March 23rd … Europe wants to send cameras, scanners and servers for registering refugees to the Sudanese regime in addition to training their border police and assisting with the construction of two camps with detention rooms for migrants."
Let’s recap that arrest warrant for the wanted, at large, alleged war criminal who is the President of Sudan, the man we are apparently hoping to pay to do our dirty work:
"Charged, as an indirect (co) perpetrator, with ten counts of crimes: five counts of crimes against humanity: murder, extermination, forcible transfer, torture, and rape: two counts of war crimes: intentionally directing attacks against a civilian population as such or against individual civilians not taking part in hostilities, and pillaging; three counts of genocide: by killing, by causing serious bodily or mental harm, and by deliberately inflicting on each target group conditions of life calculated to bring about the groups’s physical destruction, allegedly committed at least between 2003 and 2008 in Darfur, Sudan.” (my emphasis)
I mean, certainly you can criticise things like sanctions and aid conditionality, but is there any point at which we might consider it to be a bad idea to do business with this government? Is there any moral threshold here? Does Sudan need to get to be charged with 4 counts of genocide before we stop giving them stuff? Is only 5 counts of crimes against humanity not quite enough??
Note that the arrest warrant refers only to Darfur, ignoring all the other confirmed bombings of civilian and humanitarian targets by military forces in South Sudan and the Nuba Mountains, ignoring deliberately destroying crops during a famine, and ignoring the reports of slavery and child soldiers in Northern/Central Sudan.
ht: John Ashworth
12 May 2016
A fascinating abstract from Roland Fryer. In many school systems you have a single teacher for most subjects in primary school, but increasingly specialised different teachers for each subject in secondary school. This RCT tried out having subject specialist teachers in primary, finding that this worsened outcomes, in theory because at least at primary level it is more important that the teacher knows the kids’ and their ability and is therefore better able to target their teaching, than that they have more specialised knowledge of the subject matter.
Starting in the 2013-2014 school year, I conducted a randomized field experiment in fifty traditional public elementary schools in Houston, Texas designed to test the potential productivity benefits of teacher specialization in schools. Treatment schools altered their schedules to have teachers specialize in a subset of subjects in which they have demonstrated relative strength (based on value-add measures and principal observations). The average impact of teacher specialization on student achievement is -0.042 standard deviations in math and -0.034 standard deviations in reading, per year. Students enrolled in special education and those with younger teachers demonstrated marked negative results. I argue that the results are consistent with a model in which the benefits of specialization driven by sorting teachers into a subset of subjects based on comparative advantage is outweighed by inefficient pedagogy due to having fewer interactions with each student. Consistent with this, specialized teachers report providing less attention to individual students (relative to non-specialized teachers), though other mechanisms are possible.
05 May 2016
Apparently not. "Do British housing markets suffer from market failure”? The answer is no.
Here’s a quick refresher - a market failure is when the market - left alone - doesn’t produce an efficient or pareto optimal solution. Good examples are externalities such as pollution or congestion, or public goods such as new vaccines. The London housing market is not a good example of a market failure, where poor outcomes are the result of clumsy government regulation restricting supply. The fact that the market is responding to insane restrictions on new supply by focusing what little building they are allowed to do on expensive rather than affordable houses is not a market failure. Yes it is a market, and yes it is producing poor outcomes, but it is failing because of over-regulation. That is a government failure not a failure of the market mechanism.
It’s one thing when ordinary people use economic jargon in a colloquial sense with a totally different meaning to the economic term, but you expect better from a magazine called the Economist.
02 May 2016
To [Samuel] Abrams, the problem with for-profit operation of schools is not that businessmen are making money off the provision of a public service such as education. Textbook publishers, software developers, and bus operators all make money from schools and should, he said, but they are all providing a discrete good or service that can be easily evaluated. “School management, on the other hand, is a complex service that does not afford the transparency necessary for proper contract enforcement,” he said. “Without such transparency, there’s client distrust: parents, taxpayers, and legislators can never be sure the provider is doing what was promised; and the child as the immediate consumer cannot be in a position to judge the quality of service. Regular testing has been promoted as a check on quality. But teachers can teach to the test. And worse, as we know from cheating scandals in Atlanta and many other cities, teachers can change wrong answers to right answers on bubble sheets once students are done.”
From the International Education News blog at Teacher’s College, Columbia.
The trouble with the argument is that it rests on the empirical question of whether student test-based accountability systems for schools can work, and though the question is not conclusively settled, my reading is that the evidence seems to be leaning in the direction of roughly “yes".
28 April 2016
What should you read first if you’re a new policy advisor in a Ministry of Education? Here is my response to a couple of recent emails along these lines on the CGD blog.