What should you read first if you’re a new policy advisor in a Ministry of Education? Here is my response to a couple of recent emails along these lines on the CGD blog.
28 April 2016
What should you read first if you’re a new policy advisor in a Ministry of Education? Here is my response to a couple of recent emails along these lines on the CGD blog.
06 April 2016
For sure there is tacit local knowledge, but how you define “local” matters. Ken Opalo writes:
"As a social scientist, my knowledge of Kenya is largely informed by my experience as a Nairobian. Over the years I have had to learn a lot about the rest of Kenya, in much the same way an Australian would. In doing so I incurred a lower cost than a hypothetical Australian would, for sure, but the cost was not zero. And who is to say that I would necessarily be able to articulate a research agenda on whatever subject in Malawi better than a Southern Californian? What proportion of Kenyans can locate Bangui on a map?"
30 March 2016
Kishore Singh, the "UN Special Rapporteur on the right to education” is totally freaking out that apparently the government of Liberia is planning to outsource all of its primary schooling to Bridge International Academies.
“education is an essential public service and instead of supporting business in education, governments should increase the money they spend on public educational services to make them better.”
Kishore is of course commiting Mike Munger’s unicorn fallacy. The public primary school system in Liberia was frankly pretty rubbish, even before the Ebola crisis hit, and there is no evidence that just spending more money is likely to improve things.
RTI did a baseline for their early grade reading intervention project in 2008. They found that children in Grade 3 in Liberia could read 28 words per minute. To put this in perspective, in the US children are considered to be at risk if they can read fewer than 70 words per minute at the end of Grade 2. Clearly there is nothing inherently wrong with Liberian children, the schools just aren’t working (I should also be clear that I’m not beating on Liberian teachers either, it is the system that is not functioning - don’t hate the player, hate the game). The children attending free public primary schools in Liberia might be exercising a right to schooling, but they certainly aren’t exercising a right to learning or any meaningful definition of the word education.
It should be in this context that we note that AllAfrica reports that "The deal will see the Government of Liberia paying over $65 million over a five-year period.” That is $13 million per year then, to cover a full population aged 5-14 of 920,000, or roughly $14 per student per year. That’s an absolute bargain. That’s an absolute bargain, even if Bridge are only actually covering a small fraction of that full population (Normally Liberia spends around 2.5% of GDP on education, which is around $47.5 million).
Clearly the jury is still officially out on whether independent estimates will show better learning outcomes from Bridge schools, but there is a hell of a lot of research which backs up elements of their theory of change, from how teacher qualifications and experience matter so much less than the type of contract a teacher is on, to the importance of feedback for teachers on student learning.
Meanwhile over at Kishore Singh’s website, he highlights 8 key priorities, none of which include the global learning crisis, in which there are 250 million children without basic skills, the majority of whom have already sat through 4 years of school. I’m frankly baffled by the inordinate amount of hostility shown by campaigners to social entrepreneurs who are still educating absolutely tiny proportions of the global poor, and how little attention is paid to the totally failing free public systems of schooling that millions of children are enduring.
26 February 2016
"If our goal is to change behaviour and drive policies towards more effective solutions, what we have done so far is a complete failure. People who are running the What Works Clearing House don’t even have a theory [of how evidence would affect policy], or to the extent that they have a theory, its been proven wrong. … We’re just deluding ourselves if we think the 5 year, $15 million studies are having any impact whatsoever."
That’s Tom Kane (somewhat echoing Lant) on the Education Next podcast. His preferred alternative to the RCT+systematic review approach though has nothing to do with crawling on any design spaces. Rather it’s doing much more quick turn-around quasi-experimental research using the multitudes of outcomes data now being collected in the US for teacher and school accountability purposes. All that’s apparently really missing is data on the actual inputs - there is amazingly rich longitudinal data on student test scores, but no record which could be matched of what textbooks are being used in different schools, or what training courses different teachers are going on. Sounds pretty sensible to me.
At present, around 1% of Ugandans live and work overseas (roughly 400,000 of a population of 37.6 million). This 1% of the population send home 4% of Uganda’s GDP in remittances. According to a Gallup poll, around 35% of Ugandans would permanently move to another country if they were allowed to. If they all could, and sent back as much as current Ugandans abroad do, that would be a one-off 140% increase in Uganda’s economy. Or if Uganda had the same level of emigration as the UK does (8%, 5.2 million of a population of 64.1 million Brits live and work overseas), that would be a 28% increase in Uganda’s economy. And that’s totally ignoring the increase in income for the actual migrants themselves. Median income in Uganda is $2.5 per day (in PPP), so even a job on UK (“relative-") poverty pay levels of around $23 per day would be a NINE-FOLD increase for them. Needless to say, the main reason that more Ugandans don’t work in high-wage economies, is that the governments of high-wage countries impose restrictions on the entry of people, particularly those from poorer countries.
Brain drain I hear you say? Michael Clemens killed that one too (ethically it is pretty unreasonable to restrict individual's freedom to such an extreme extent, even if there were any real evidence that emigration of highly-skilled people hurt an economy, which there isn't anyway).
So yes, clearly planning Kampala’s urban development is important for growth. But does it really have more “potential” than something that could double the country’s economy overnight?
26 January 2016
Courtney Martin has an interesting post at the Development Set about the “reductive seduction” of other people’s problems. Problems we know something about (gun control in America as her example for the Americans) seem complex, political, and intractable, whereas problems we know less about (rural hunger or girl’s secondary education or homophobia in Uganda) seem more straightforward.
Which reminds me a little of the difference between statements from leaders on education in their own country and in other people’s countries.
Here’s Julia Gillard on education in Australia:
"We need a commitment to transparency and accountability. It's my strong view that lack of transparency both hides failure and helps us ignore it. It feeds a culture where all the adults involved – the teachers, the principals, the community leaders and the members of parliament – avoid accountability. And lack of transparency prevents us from identifying where greater effort and investment are needed. Importantly, transparency and accountability are overwhelmingly supported by parents."
And Julia Gillard on education in developing countries:
Constantly improving ways of working.
An even greater embrace of the power of partnership."
"we will focus on classroom standards, ensure that we monitor exam standards rigorously, reform the qualifications system … World class education, we know, achieves high standards for 100% of the children when there are systems of accountability, funding and pupil tracking that leave no child behind and personalised learning is tailored to the unique potential of every child with one-to-one tutoring and support. That world class education depends on a systematic intolerance of failure and a preparedness of public authorities to intervene and to innovate to eradicate failure"
"The biggest obstacle to what would be a spectacular achievement – as is so often the case – is a shortage of money."
David Halpern, Chief Exec at the Behavioural Insights Team (aka “Nudge Unit”) has a new blog up about how behavioural science can be used to improve people’s life chances.
"Why is it that a Kenyan market seller spends half her profit on money lenders rather than saving a tiny sum each day to escape such debt? Or why does a low income family in the UK or USA spend twice as much on a stove (cooker), bought on expensive hire purchase, than a middle class family?"
The thing is, as we know from Branko Milanovic, the country that you are born in matters more for your life chances than everything else combined. "A proper analysis of global inequality today requires an empirical and mental shift from concerns with class to concerns with location,"
So what does the BIT have to say about the movement of people? As Matt points out, the Home Office is currently paying the BIT to find ways to convince illegal migrants to voluntarily leave the UK. That is, to support the Home Office in its agenda of shutting down the single best way that exists of improving someone’s life chances. Perhaps someone could instead work on a clever “nudge” to make people less scared of foreigners?
21 January 2016
This post was first published on the CGD Views from the Center Blog
Uganda goes to the polls in 30 days to elect its next president, but there is little sign so far in the public debate on education of the need to shift focus from inputs and enrolment to actual learning outcomes.
I was in Kampala last week piloting a survey on school management (more on that later), and spotted in the Daily Monitor a feature on the candidate’s campaign promises on education, reading as follows:
- One primary school per parish (to reduce average walking distances)
- Continue to increase the budget allocation for text books
- Introduce compulsory universal primary education
- Increase remuneration for primary school teachers
- Recruit and train new teachers with the aim of reducing the teacher-student ratio
- Build more schools and classrooms
That’s zero mention of actual student learning outcomes from any of the leading candidates, and a complete focus on spending more money and providing more of the inputs that have been showntime and again to bear little relationship with improved learning outcomes.
NYU Professor David Stasavage published a paper in 2005 exploring how the introduction of elections in Uganda in 1996 helped lead to the removal of school fees in 1997. He also published a follow-up in 2013 noting how elections focus politicians on those things that are easily visible to voters. Fees for tuition at public schools are very visible to voters, and so one of the first things democratic politicians address. School quality is much less visible to the average voter, leading to much less focus on teaching and learning by politicians.
All of this suggests that one way to improve student learning is to get citizens and politicians more focused on learning by better measurement and spreading of the insight that despite high enrolment, student skills are very poor. This is a key part of the theory of change behind the global ASER/Uwezo/PAL-Network movement of citizen-led student reading assessments. What sadly seems clear from Uganda is that this message has not yet got through. We’ve known since Uwezo’s 2010 assessment that children in Uganda are way behind where there should be (only 2 percent of grade 3 children could read and understand a grade 2 story).
My tip for anyone with the opportunity to grill the candidates on education policy would be to borrow Paul Atherton’s mantra: “But can the kids read?”
20 January 2016
There’s a strong argument for “made” from Elizabeth Green.
"Russ: But there is a view out there, and you talk about it at some length in the book, that some people believe great teachers are just born and not made. And that there is a certain 'it' quality that teachers have that make them more effective in the classroom in all kinds of dimensions. What do you think of that argument, and why is it an important argument in the debate?
Guest: I think that that argument is embedded in the way we talk about education policy, teacher policy. We say, there are good teachers and then there are bad teachers, and then what we need to do is either find more of the good teachers, people who are destined to become good, by doing a better job of recruiting good teachers. Or, we need to incentivize good teachers to stay, or we need to create better, easier, more effective ways to remove bad teachers from the classroom. And I think that what that construct is built on is, as you say, this assumption that teaching quality is something that's natural born in people--that it's about personality traits or character traits. But in fact every research study that's tried to connect character traits and personality traits to who becomes an effective teacher fails to find that any of them make any difference. So, an extrovert or an introvert doesn't matter for how effective you'll be in the classroom. So, I think that what instead is more convincing to me for what matters is what teachers do, and what they know. And that's very different from a natural born trait, something that you need to learn.
Russ: So, we're going to talk about Doug Lemov, who was a guest here on EconTalk. He plays a large role in your book. But one of the things he emphasizes, of course, is practice. So, one view says the reason we don't have better teachers is they don't practice. What do you think of that argument?
Guest: Yeah. So, Doug is obviously, for people who listen to your show, they know that he's a former teacher who became the leader of a group of schools called the Uncommon Schools network. And he encountered the same realization, that what he called the 'Build it, Buy it' problem. So, at first, early on, he tried to improve the quality of teaching in schools by buying teachers who are already good. But over time that became unsustainable, and he realized he had to help build good teachers from all of--any person that he could recruit. So, he couldn't just recruit his way to excellence; he had to build it"
23 December 2015
So despite not believing, I’ve always been curious about religion. Clearly most human beings do believe in something supernatural, which is interesting and worthy of some thought. I was struck a few years ago by Alain de Botton’s book “The Consolations of Philosophy” how my modern liberal ethics and values are not in fact rootless, but deeply rooted in centuries of philosophy, and how us modern liberals are missing something that the church provides - people who’s job it is to be a kind of very practical applied philosopher, translating all this history and helping people to live better and cope with difficulty (Alain then made this argument himself explicitly in a follow-up book “Religion for Atheists”). I’ve also been struck by the economics research on happiness, which finds that people who practice religion tend to be on average happier, and that this is probably partly due simply to turning up to the same place every week with the same group of people, and partly something to do with the actual content - the consolation of the philosophy.
So my very rational calculating mind has turned further and further away from my earlier extremist Dawkins-ist atheism-ism (I don’t like religion because too many religious teachings are homophobic) towards greater curiosity with things like de Botton's "School of Life”, "the Sunday Assembly”, and modern Stoicism. My granddad actually even had a “humanist” funeral (I knew he was not religious but had no idea that he was so actively committed to non-religion), so this strain of thinking clearly runs in the family.
Then one day a couple of months ago I walked past the sign outside the church on Newington Green near where I live, proclaiming itself “The Birthplace of Feminism”, and remembered to look it up when I got home. And so as it turns out, this is the church that Mary Wollstonecraft attended in 1792 when she published “A Vindication of the Rights of Woman”, one of the first works of feminist philosophy, well over a century before the Suffragettes campaigned for votes for women. Pretty cool. And then it turns out, the current Minister of the church, Andy Pakula, is an atheist. An atheist with a PhD in biology from MIT who loves science. He writes: "As a scientist, I would not entertain any ideas that could not be proven in a well-designed, objective experiment.” This is a church that stopped carrying out marriages in 2008 on principle until all couples had equal marriage rights.
And so, living 5 minutes walk away, I really had no excuse not to go. And it’s great, a place of “radical inclusivity”, where you are "welcome whether you are female, male, or other”, or anything else, and people come together to sing and listen to readings and share personal struggles and sit in silence.
This last Sunday before Christmas we had a very appropriately themed set of nativity stories (about weary travellers being told that there’s no room at the inn), seamlessly mixed with a heavy dose of amazing current stories, from a member of the congregation who just got back from spending a week helping and welcoming refugees landing on a Greek island, and from the church's charity of the month, which is a social club named Akwaaba (Twi for “welcome") in Hackey, for refugees, asylum seekers, and migrants. We’ve actually got loads of room at the inn (in fact the inn is facing a massive aging population crisis such that we need millions of working aged people to come live in the inn if we’re going to have any hope of paying for all those pensions), so it’s nice to be around people who appreciate that too.
So to conclude; research suggests that participating in an organised religion makes you happier, and if you’re a woolly secular liberal like me, then there are churches out there that are secular, actively pro- gay rights, pro- migrant rights, and radically inclusive.
If you fancy taking a look, New Unity has a blog, a livestream on Sunday mornings, and a Youtube channel, but obviously none of these compare to being there in person, and there’s also a whole worldwide Unitarian/Universalist church movement (though it’s possible that not all the other churches will have atheist and MIT-trained scientist Ministers).
16 December 2015
It might seem obvious to some of you reading this that it might be possible to learn something from a book. But as a recent review for RISE by Paul Glewwe and Karthik Muralidharan found, researchers have actually so far failed to show rigorously that there is any improvement in test scores in developing countries after handing out textbooks to schools. There have now been four different Randomized Controlled Trials showing no improvement (and for four different reasons).
So when might books “work”? A new paper from the World Bank proclaims to answer just this question: “When Do In-service Teacher Training and Books Improve Student Achievement? Experimental Evidence from Mongolia.”
Their answer, somewhat disappointingly, seems to be “when it happens in Mongolia”. More constructively, though books alone seem to work, they work better when combined with teacher training, reinforcing Glewwe and Muralidharan’s conclusion about the importance of complementarities.
But that “there are complementarities” isn’t a very satisfying conclusion by itself. The more comprehensive hypothesis being developed for RISE by Lant Pritchett is: “when there is an accountability framework which is coherent for learning” – that is, when all of the relevant actors are held accountable for common goals through clear delegation of those goals, and have the resources to accomplish them. We’re hoping that this accountability coherency system diagnostic can be a useful tool for thinking through systematically what it is about specific contexts that mean that interventions work in some places and not others. What is it about Mongolia which means that providing books alone can be enough, in contrast to those other studies in Kenya and Sierra Leone? It just might be a coherent accountability system.
23 November 2015
A couple of weeks ago, France24 ran a story featuring accusations by Belgian Professor Filip Reyntjens that the Government of Rwanda had manipulated its poverty statistics. The truth, to my relief*, is somewhat less exciting.
What seems to have actually happened, is that Rwanda quite resonably decided to update the methodology for calculating what the poverty line should be, but then found that the new methodology led to an implausibly high poverty line, and so decided to (slightly arbitrarily) “adjust” the new methodology, resulting in the final poverty line being almost exactly what you would have expected it to be had you simply updated the original poverty line for inflation.
It took me a while to figure all this out, as the original criticism and rebuttal by NISR weren’t entirely clear, and it was only in Filip’s reaction to NISR’s rebuttal that I grasped his (mistaken) point (here’s also the Rwanda EICV4 Report and EICV3 Report).
How is poverty measured?
Rwanda has followed a fairly typical process – set a poverty line by first defining a minimum quantity of calories needed, second working out how much it would cost a poor person to buy that many calories, third increasing that amount by 40% to account for some basic minimum non-food spending needs. Then to get your poverty rate, just calculate how many people spend less than the poverty line.
What was the disagreement about?
Rwanda’s poverty line was set in 2001 based on how much it cost then to purchase a basket of goods that poor people bought back then. You need to keep your methodology consistent over time to allow for fair comparisons, but its also reasonable to think that the minimum consumption basket is likely to change over 15 years of rapid growth.
The government of Rwanda decided to keep the minimum assumed number of calories (2,500 per day, which is pretty high), but change step 2 – the way of working out how much it costs to buy these calories. In a normal survey year, this cost is simply updated for inflation (even if prices and consumption habits have changed in the meantime). This year, Rwanda decided to make an update to the prices and consumption habits, but found something odd. Most poor people consume far fewer than the minimum number of calories – almost half. So how do you construct a hypothetical “ bare minimum" food consumption basket, that is twice as big as what people actually buy? Do you just double everything? Or do you assume that if people bought more food than they did, they might buy more of some items than others? This is where the big disagreement presumably came. Rather than choosing to simply double everything, the Rwandan stats agency made a few arguably arbitrary choices about which items to increase and which to decrease, that has a big effect on the overall price of the basket, and therefore the overall poverty line, and therefore the poverty rate.
Why is Filip Reyntjens wrong?
Filip argues, correctly, that Rwanda’s assumptions about how to scale up consumption patterns to reach their minimum calorie basket, affects the overall line. In fact, their adjustments reduce the line by 19%. But his next step is wrong. He argues that as this new methodology line has been reduced by 19%, we should also reduce the 2010/11 line by 19%, giving a substantially lower poverty rate in 2010/11, and therefore an increase in 2013/14. But he misses the intermediate step – the Rwandans didn’t just adjust the new food basket, they first also calculated a whole new food basket.
Yes, what should really happen is for the new methodology to applied retrospectively to all the old survey data to allow for truly comparable numbers, but the adjustment made to the new methodology leads you to a poverty line that is basically the same as it would have been with the old methodology anyway.
Implications for how we measure poverty?
One thing that this choice really highlights is the number of assumptions you sometimes need to make, and the fragility of the whole concept of poverty estimates.
Here's an example of another seemingly arbitrary choice of assumption with big consequences – the Indian stats agency used to measure poverty with surveys that asked people how much food they had bought in the last 30 days, longer than practice elsewhere which uses 7 day recall periods. To their great credit the stats agency decided to run a randomised control trial to test these two methods against each other. The result was that moving from a 30 day to a 7 day recall period increased measured consumption massively – and reducing poverty by 175 million people – close to half of all those in poverty (from Angus Deaton’s 2014 LSE lecture, via Nic Spaull).
The bottom line: measurement is hard, and it is possible for reasonable people to disagree, without there necessarily being any nefarious trickery.
* Relief, because I have previously worked both for OPM as a staff member, on a project with OPM for the Rwandan Stats agency, and directly on a project for the Rwandan Ministry of Finance.
17 November 2015
How do you compare the good that the UK is doing with its whopping 0.7% aid budget, against the good that Germany is doing by accepting large numbers of refugees? A smart (German) friend asked me if there are any numbers on the size of the remittances we might expect to see from Syrian refugees in Germany to Syria. Of course, remittances are far from the most important reason for accepting refugees, but they do allow for a nice easy cash sum with which we can make a comparison to aid flows.
But in terms of numbers of refugees, Germany expects to take 800,000 this year (compared to just a few thousand in the UK), though fewer than that have been documented so far, and not all will be Syrian. Let’s assume for a moment that the total will be 400,000 from Syria, and they will be quickly processed so that they are able to work. If every Syrian refugee in Germany was able to send home £1,000 to family and friends, that already equal Britain’s aid budget for Syria. Is £1,000 a realistic prospect? One way to think about this is to look at remittances from existing migrants in Germany (p33) to the middle east. There are currently around 67,000 migrants from Lebanon living in Germany, who send back to Lebanon almost $1 billion a year - that’s around £9,500 each, which seems almost implausibly large, but who knows, the Chinese and Vietnamese also send home large sums, and the Nigerians send home even more. In any case, it certainly seems plausible, even likely, that Syrian refugees to Germany, once permitted to work for even low German salaries, will be able to send home at least £1,000, if not more.
12 November 2015
Duncan Green reviews a fascinating new AidData survey on what developing country policymakers think about donors.
One of the key findings he points to is that
"Reliance upon technical assistance undermines a development partner’s ability to shape and implement host government reform efforts. The share of official development assistance (ODA) allocated to technical assistance is negatively correlated with all three indicators of development partner performance."
Obviously alarm-bells should be ringing about such firm causal conclusions being drawn from a correlation. One of the best ways of assessing these things is with some rigorous eyeball econometrics - take a look at this chart showing the relationship driving that claim.
Looks to me like that is a pretty weak relationship, and you could just as easily have drawn a totally flat line (no relationship). And indeed, deep in the weeds, Table E.11 tell us that this is a simple correlation between these two variables with a sample size of just 44 data points (countries). It might technically pass a statistical significance test, but it doesn’t really tell us that there is a reliable correlation, let alone causality. And even if you believed the estimated negative relationship - it’s really not huge - implicitly going from 0% aid on technical assistance to a massive 50% of aid spent on technical assistance would only reduce the perceived quality of your advice by 0.55 points on a 5 point scale.
Bottom line for technical assisters - don’t give up your day job quite yet.