23 November 2015

No, Rwanda didn't "fiddle" its poverty stats

A couple of weeks ago, France24 ran a story featuring accusations by Belgian Professor Filip Reyntjens that the Government of Rwanda had manipulated its poverty statistics. The truth, to my relief*, is somewhat less exciting.

What seems to have actually happened, is that Rwanda quite resonably decided to update the methodology for calculating what the poverty line should be, but then found that the new methodology led to an implausibly high poverty line, and so decided to (slightly arbitrarily) “adjust” the new methodology, resulting in the final poverty line being almost exactly what you would have expected it to be had you simply updated the original poverty line for inflation.

It took me a while to figure all this out, as the original criticism and rebuttal by NISR weren’t entirely clear, and it was only in Filip’s reaction to NISR’s rebuttal that I grasped his (mistaken) point (here’s also the Rwanda EICV4 Report and EICV3 Report).

How is poverty measured?

Rwanda has followed a fairly typical process – set a poverty line by first defining a minimum quantity of calories needed, second working out how much it would cost a poor person to buy that many calories, third increasing that amount by 40% to account for some basic minimum non-food spending needs. Then to get your poverty rate, just calculate how many people spend less than the poverty line.

What was the disagreement about?

Rwanda’s poverty line was set in 2001 based on how much it cost then to purchase a basket of goods that poor people bought back then. You need to keep your methodology consistent over time to allow for fair comparisons, but its also reasonable to think that the minimum consumption basket is likely to change over 15 years of rapid growth.

The government of Rwanda decided to keep the minimum assumed number of calories (2,500 per day, which is pretty high), but change step 2 – the way of working out how much it costs to buy these calories. In a normal survey year, this cost is simply updated for inflation (even if prices and consumption habits have changed in the meantime). This year, Rwanda decided to make an update to the prices and consumption habits, but found something odd. Most poor people consume far fewer than the minimum number of calories – almost half. So how do you construct a hypothetical “ bare minimum" food consumption basket, that is twice as big as what people actually buy? Do you just double everything? Or do you assume that if people bought more food than they did, they might buy more of some items than others? This is where the big disagreement presumably came. Rather than choosing to simply double everything, the Rwandan stats agency made a few arguably arbitrary choices about which items to increase and which to decrease, that has a big effect on the overall price of the basket, and therefore the overall poverty line, and therefore the poverty rate.

Why is Filip Reyntjens wrong?

Filip argues, correctly, that Rwanda’s assumptions about how to scale up consumption patterns to reach their minimum calorie basket, affects the overall line. In fact, their adjustments reduce the line by 19%. But his next step is wrong. He argues that as this new methodology line has been reduced by 19%, we should also reduce the 2010/11 line by 19%, giving a substantially lower poverty rate in 2010/11, and therefore an increase in 2013/14. But he misses the intermediate step – the Rwandans didn’t just adjust the new food basket, they first also calculated a whole new food basket.

Yes, what should really happen is for the new methodology to applied retrospectively to all the old survey data to allow for truly comparable numbers, but the adjustment made to the new methodology leads you to a poverty line that is basically the same as it would have been with the old methodology anyway.

Implications for how we measure poverty?

One thing that this choice really highlights is the number of assumptions you sometimes need to make, and the fragility of the whole concept of poverty estimates.

Here's an example of another seemingly arbitrary choice of assumption with big consequences – the Indian stats agency used to measure poverty with surveys that asked people how much food they had bought in the last 30 days, longer than practice elsewhere which uses 7 day recall periods. To their great credit the stats agency decided to run a randomised control trial to test these two methods against each other. The result was that moving from a 30 day to a 7 day recall period increased measured consumption massively – and reducing poverty by 175 million people – close to half of all those in poverty (from Angus Deaton’s 2014 LSE lecture, via Nic Spaull).

The bottom line: measurement is hard, and it is possible for reasonable people to disagree, without there necessarily being any nefarious trickery.


* Relief, because I have previously worked both for OPM as a staff member, on a project with OPM for the Rwandan Stats agency, and directly on a project for the Rwandan Ministry of Finance.

17 November 2015

Why Germany is probably doing more for Syria than the UK

How do you compare the good that the UK is doing with its whopping 0.7% aid budget, against the good that Germany is doing by accepting large numbers of refugees? A smart (German) friend asked me if there are any numbers on the size of the remittances we might expect to see from Syrian refugees in Germany to Syria. Of course, remittances are far from the most important reason for accepting refugees, but they do allow for a nice easy cash sum with which we can make a comparison to aid flows.

The UK is spending somewhere between £200 million and £400 million on Syria this year. For comparison, whilst Germany is ramping up aid spending, it is still less than 0.4% of GDP overall.

But in terms of numbers of refugees, Germany expects to take 800,000 this year (compared to just a few thousand in the UK), though fewer than that have been documented so far, and not all will be Syrian. Let’s assume for a moment that the total will be 400,000 from Syria, and they will be quickly processed so that they are able to work. If every Syrian refugee in Germany was able to send home £1,000 to family and friends, that already equal Britain’s aid budget for Syria. Is £1,000 a realistic prospect? One way to think about this is to look at remittances from existing migrants in Germany (p33) to the middle east. There are currently around 67,000 migrants from Lebanon living in Germany, who send back to Lebanon almost $1 billion a year - that’s around £9,500 each, which seems almost implausibly large, but who knows, the Chinese and Vietnamese also send home large sums, and the Nigerians send home even more. In any case, it certainly seems plausible, even likely, that Syrian refugees to Germany, once permitted to work for even low German salaries, will be able to send home at least £1,000, if not more.

12 November 2015

Is “technical assistance” counterproductive?

Duncan Green reviews a fascinating new AidData survey on what developing country policymakers think about donors.

One of the key findings he points to is that

"Reliance upon technical assistance undermines a development partner’s ability to shape and implement host government reform efforts. The share of official development assistance (ODA) allocated to technical assistance is negatively correlated with all three indicators of development partner performance."

Obviously alarm-bells should be ringing about such firm causal conclusions being drawn from a correlation. One of the best ways of assessing these things is with some rigorous eyeball econometrics - take a look at this chart showing the relationship driving that claim.

Looks to me like that is a pretty weak relationship, and you could just as easily have drawn a totally flat line (no relationship). And indeed, deep in the weeds, Table E.11 tell us that this is a simple correlation between these two variables with a sample size of just 44 data points (countries). It might technically pass a statistical significance test, but it doesn’t really tell us that there is a reliable correlation, let alone causality. And even if you believed the estimated negative relationship - it’s really not huge - implicitly going from 0% aid on technical assistance to a massive 50% of aid spent on technical assistance would only reduce the perceived quality of your advice by 0.55 points on a 5 point scale.

Bottom line for technical assisters - don’t give up your day job quite yet.

02 November 2015

Why are people so opposed to immigration? #142538

As the evidence piles up that migrants don’t steal jobs (one of the implications of them being human beings is that migrants also buy stuff - so they create exactly as many new jobs as they “take”), some of the more sophisticated immigration opponents turn to the negative impacts of immigration on other things such as housing or public services instead to support their case.

So what does the research evidence say about the impacts of immigration on public services? Really very little actually. The University of Oxford’s Migration Observatory says that there is “no systematic data or analysis.” In health, we know that many healthcare providers are immigrants, but it’s hard to know the impact of migrants as users of health services as (rightly) nobody records people’s migration status when they go to the doctor.

Using household survey data, Jonathan Wadsworth at Royal Holloway found that (shock!) immigrants tend to use GP services and hospitals at roughly the same rate as natives (via Ferdinando Giugliano in the FT).

Taking another approach, a new paper by Osea Giuntella from the Blavatnik School of Government at Oxford, combines household survey data with administrative data on NHS waiting times. Do you need to wait longer for a referral or in A&E in places where there are more immigrants? Come find out at the CGD Europe research seminar on Weds 18 Nov (there will be sandwiches).

28 October 2015

First RISE Working Papers

The first set of working papers from RISE (Research on Improving Systems of Education) are out.

Paul Glewwe and Karthik Muralidharan have an updated review of everything we know about rigorous evaluations of interventions to improve learning in developing countries (paper here, my comment on the RISE blog here).

Rukmini Banerji describes how a disruptive pedagogical innovation spread (and didn’t) in Bihar, and Kara Hanson tell us what education can learn from health systems research.

Mari Oye also has a blogpost up about the UN Myworld survey and the SDGs.

Coming soon, Lant’s consolidated explanation of what an Education System actually is, grand general theory of why some things work sometimes but not all the time, and tentative framework for diagnosing systems for constraints and prioritising action. Watch this space.

09 October 2015

We have no idea what countries are spending on education

Listen to some international education people and you get the impression that the education problem is mostly solved if we could just spend more money. The story goes something like “Poor countries spend X on education, if they could 1.5X then all the kids could get a good education, they can’t afford 1.5X, so we should fill the gap with aid."

The reality is, even if it was the case that just filling the gap would solve the problem (which is dubious to say the least) , we don’t really even know what the gap is.

This is Silvia Montoya, Director of the UNESCO Institute for Statistics:
"governments need detailed and disaggregated data to ensure that their resources are allocated equitably and effectively within their education systems. At the same time, donors need the data to better evaluate whether the aid they provide is an incentive for governments to increase spending commitments or if they are crowding-out domestic resources.

For the moment, the availability and completeness of education finance data is unfit for these purposes, with less than one-half of countries able to regularly report key information, such as total government expenditure on education” (my emphasis)
Nevermind the purpose of accountability and transparency to the citizens of developing countries...

Good luck to the new Commission on Financing Global Education!

Does foreign aid harm political institutions?

Good news for reflective aid business -types who like agonising about what the point of it all is and sometimes wondering whether we’re even making things worse (err... talking about a friend... not me...). Also even good news for developing countries I suppose.

A new paper in the Journal of Development Economics by Sam Jones & Finn Tarp* using new data on aid (from aiddata.org) and institutions (from the Quality of Government Institute) finds no evidence that aid has undermined institutions on average, if anything there seems to be a positive relationship. I’m less confident in the positive findings than reassured that in *none* of their various different approaches is the relationship negative.

Now you’re probably thinking “what about the 2006 CGD review paper by Todd Moss, Gunilla Pettersson & Nicolas Van de Walle, described by Blattman as "the best summary I know of the evidence”, which concluded that aid could have a harmful effect on institutional development”? Well the word “could” is important there - that conclusion was somewhat speculative, and this new evidence from Jones & Tarp fills an important gap in terms of systematic quantitative evidence on this topic, and should probably shift your priors at least a little in that direction.

I wonder what Angus Deaton would say?


* Thanks to UNU-WIDER the paper is open-access, which is great for what it is, but obviously having public institutions pay private journal owners something greater than the cost of production isn’t an ideal long-run equilibrium, and we really need something that fundamentally shifts the whole publishing industry.

02 October 2015

The State of the Humanitarian Aid System 2015

“ALNAP” launched today the 2015 “State of the Humanitarian Aid System” Report.

One of the key findings highlighted in their fancy infographics:
"44% of aid recipients surveyed were not consulted on their needs by aid agencies prior to the start of their programmes”.
In totally unrelated news, the DFID-ODI-CGD High Level Panel on Humanitarian Cash Transfers chaired by Owen Barder published it’s report a few weeks ago, arguing that much more use should be made of cash transfers, because most of the time they are more cost effective than giving out stuff.

In further totally unrelated news, DFID published two press releases today highlighting substantial non-cash aid in response to humanitarian crises in the Central African Republic and Malawi.

In Owen’s words: "the questions should always be asked: “Why not cash? And, if not now, when?”"

21 August 2015

Is rescuing migrants from the Med good value for money?

Thousands of people die each year trying to cross the Mediterranean to seek asylum in Europe. Christopher and Regina Catrambone, American and Italian entrepreneurs, decided to take matters into their own hands and set up their own private rescue mission.

Naturally when I read that

"a fundraising drive by the activist organisation Avaaz reached $500,000, slightly less than a month’s costs”,

I started wondering about cost effectiveness. Elsewhere the Guardian article states

"Setting up Moas was not cheap, with monthly operating costs of up to €600,000"


"The Phoenix rescued 1,462 people in 10 weeks"

So let’s go with the higher figure of €600,000 per month - over 10 weeks (2.3 months) that is a total cost of €1.4m (roughly $1.6m or £1m). And to save 1,462 people, that is a cost of £700 (~ $1000) per death averted.

Is that a lot or a little? As Owen has pointed out, the UK NHS considers anything less than £100,000 per death averted to be good value for money.

At the other extreme, childhood vaccinations, "long recognized as among the most cost-effective uses of limited health resources in low-income countries” (Disease Control Priorities) cost $275 per death averted.

At face value, Catrambone’s "Migrant Offshore Aid Station" (MOAS) looks like a pretty good value for money philanthropic bet.

04 August 2015

Effective Altruism, RCTs, NGOs, & the Government End-Game

Good Ventures just gave a $25 million unrestricted grant to Give Directly on the advice of Givewell. That’s a lot of good news in one sentence, but it’s not even the best part. Givewell buried the lede when they mention around paragraph 20 that;

"GiveDirectly plans to discuss partnerships with the following types of institutions:

- Donor aid agencies
- Developing country governments (national and local). (For example, several governors in Kenya have already approached GiveDirectly about running cash transfer programs in their counties.)"

That’s what it’s all about. To really get sustainability and scale in social policy you need government involvement - that’s why the best NGOs combine a mixture of immediate direct service delivery in places where government just doesn’t have the capacity to deliver, with support to interested governments to build that capacity for the longer-term, often at the local level where administrators struggle to actually implement well-designed central policy documents, and with innovation in new models of service delivery, that governments might later adopt, of which GiveDirectly is clearly a strong example. Similarly whilst Innovations for Poverty Action and J-PAL may have started off following that recently infamous Kremer-Miguel deworming study by working on service delivery through small NGOs, their focus is on things that can work at scale, and having built a reputation through working with NGOs have been able to transition to working with governments (for example in Ghana and Peru).

As Jessica Brass writes,

"Government and NGOs learn from each other to improve what they do. In particular, many government agencies notice the successes achieved by NGOs and, whether intentionally or not, mimic their actions"

So yes, maybe some of the effective altruists can be accused of being philosophers not development wonks, and potentially even naive about politics, but for every anecdote-backed theoretical case for how aid might undermine the process of building citizen-state accountability, I can come up with an anecdote-backed theoretical case for how aid can support improved governance through innovation in service delivery models, and until we get some quantitative evidence on the issue, I don’t see how else we’re going to resolve the debate.

Did I miss anything?

23 July 2015

New education economics papers

A few papers caught my eye from last month's repec new education economics papers feed. All from developed countries, but such is economics, a lot of the interesting new research happens on rich countries where the researchers are more likely to know about interesting policies and institutional features to study, and where there is better data (both problems which RISE is seeking to address, by encouraging collaborations between developing country-based researchers and leading academics based at top universities in rich countries, and also by funding new data collection in developing countries).

"Quantifying the Supply Response of Private Schools to Public Policies” by Michael Dinerstein and Troy Smith looks at a reform in New York which increased the budget for some public schools, finding an increase in enrolment at these schools, and that nearby private schools lost business and were slightly more likely to shut down. In an interesting twist, whilst the reform improved quality at the public schools that received extra money, the movement of some students from higher quality private schools to lower quality public schools meant that overall outcomes from the school system were not improved. All of which reminds me of the recent story from Rwanda that some private schools seem to be going out of business by the growth of public schools. What is that shift doing to the overall quality mix?

The Information Value of Central School Exams” by Guido Schwerdt & Ludger Woessmann compares students in Germany who graduated from states which use a centralized common school-leaving exam to those with a local school-set leaving exam. Better grades are roughly three times more valuable in the labour market when they come from centralized exams than from school-set exams. In Lagos, private school associations are currently in the process of joining together to put their students through common school leaving exams for partly this reason.

Nicola Bianchi’s Job Market Paper looks at "The General Equilibrium Effects of Educational Expansion” - when Italy expanded STEM higher education in 1961, enrolment increased by 200%. However - those students who enrolled didn’t earn any more than they would have had they not enrolled, because the massive increase in the supply of qualified students reduced the labour market premium for that qualification, as well as the quality of education suffering due to congestion and peer effects. Which of course should remind you of Lant’s classic “Where has all the education gone?"

How much does the new deworming replication matter for Effective Altruists?

It doesn’t at all, as far as I can tell. As Calum points out, what matters is the systematic review of evidence not one study. And the new Cochrane systematic review doesn’t seem to have responded to the criticism from Duflo et al to their 2012 review, that it ignores quasi-experimental and long-term evidence on positive impacts of deworming (specifically Bleakley 2004, Ozier, and Baird et al).

A replication of the famous Miguel and Kremer deworming paper that launched the whole RCT in development economics movement, is published in the Journal of International Epidemiology today (along with comment from Hicks, Kremer, and Miguel, and reply from the replication authors), with coverage in the Guardian and by Ben Goldacre for Buzzfeed.

You may remember Berk Ozler's review of the draft of the replication paper back in January - concluding

"Bottom line: Based on what I have seen in the reanalysis study by DAHH and the response by HKM, my view of the original study is more or less unchanged."

You can probably expect to see more on the replication coming from @cblatts, which I’m not going to get into, but back in 2012, Givewell were convinced that the Cochrane review shoudn’t change their recommendation to donate to the Schistosomiasis Control Initiative or Deworm the World.

The ambiguity does make me a little queasy, and pushes me more in the direction of GiveDirectly (I see basically zero risk that giving $1000 to someone on a very low income can really be totally wasted, in the way that an ineffective drug could theoretically have zero impact).

16 July 2015

Migration fact of the day

"Today, approximately 7 million Indians work in six GCC countries, which is more than 50% of estimated 13 million foreign workers present in the GCC. The Indian workers in GCC remit about US$40 billion i.e. around 57% of the total remittances, i.e. US$70 billion India receives annually. Besides contributing significantly to the national forex reserves, the remittances received directly by the workers’ families help in poverty alleviation, support local business, promote entrepreneurship and generate employment."

That’s Zakir Hussain on the World Bank blog. Worth remembering this context next time you read a scandal about the poor treatment of Indian workers in the Gulf.

15 July 2015

Calling Education Researchers…

I just got back from the fourth of seven events being held around the world drumming up interest in bidding for the RISE “Research on Systems of Education” project. There is £21 million of DFID money to be split between 5 country research teams (with a preference for bids from Ethiopia, Nigeria, Ghana Pakistan, India, Indonesia, Vietnam, or other DFID focus countries) to study reforms that are happening to education systems that might credibly have a big impact of student learning. EOIs due 23rd August. 
There is plenty more information on the CGD website and the new RISE website,  but to make things really easy, here are a few key links about the project and how to bid (very helpfully put together by Mari).