How much does the new deworming replication matter for Effective Altruists?

It doesn’t at all, as far as I can tell. As Calum points out, what matters is the systematic review of evidence not one study. And the new Cochrane systematic review doesn’t seem to have responded to the criticism from Duflo et al to their 2012 review, that it ignores quasi-experimental and long-term evidence on positive impacts of deworming (specifically Bleakley 2004, Ozier, and Baird et al).

A replication of the famous Miguel and Kremer deworming paper that launched the whole RCT in development economics movement, is published in the Journal of International Epidemiology today (along with comment from Hicks, Kremer, and Miguel, and reply from the replication authors), with coverage in the Guardian and by Ben Goldacre for Buzzfeed.

You may remember Berk Ozler's review of the draft of the replication paper back in January - concluding

"Bottom line: Based on what I have seen in the reanalysis study by DAHH and the response by HKM, my view of the original study is more or less unchanged."

You can probably expect to see more on the replication coming from @cblatts, which I’m not going to get into, but back in 2012, Givewell were convinced that the Cochrane review shoudn’t change their recommendation to donate to the Schistosomiasis Control Initiative or Deworm the World.

The ambiguity does make me a little queasy, and pushes me more in the direction of GiveDirectly (I see basically zero risk that giving $1000 to someone on a very low income can really be totally wasted, in the way that an ineffective drug could theoretically have zero impact).

Evidence on global education: A lit review in one chart

From Stefan Dercon's presentation at the recent "Town Hall" event on funding opportunities for international education research. He explains further in this blog post. Other presentations from representatives from the World Bank, USAID, and ESRC, are available here. 

The results agenda is yet to take hold in the UK

DFID Annual Budget: £10 billion

Current (domestic) UK Government "Major projects expenditure" with no plans to evaluate impact or value for money: £49 billion (NAO 2013: Evaluation in Government)

Two ways to make the world a better place

According to Angus Deaton, either be like Jean Drèze, or be like CGD. He actually comes off pretty well in this interview.

"The moral obligation is important because I don’t want it to sound like I’m a heartless bastard who has no interest in this partly because there’s just this: these people are hurting and if you can help them you ought to help them. Secondly, some of their hurt is to do with us, you know the colonial programme was not a great success. It might have been a great success for the Brits, it was not a great success for what happened in India. So we owe them big.

I have students I meet at Princeton who come to me and say “I want to devote my life to making the world a better place” and “I want to dedicate my self to reducing global poverty” and I say there are two ways: one is impossibly hard but I know at least one person who did that, some other people have done it. You go to Sierra Leone, you go to India or wherever. You become a citizen, you use your skills to help local groups agitate. You don’t take any money from outside, you just become like them and you use the skills and knowledge you’ve learnt here to help them. My friend Jean Drèze is an activist in India who’s been incredibly successful in doing this. He had to renounce his Belgian citizenship, it was very hard for him to even get that done. He lives without money because he’s frightened of being compromised by that and he’s been enormously successful. But it’s like the camel going through the eye of the needle right? It’s hard. 

The other thing I tell my students to do is go to Washington and tell them to stop selling arms to poor countries. These are very very articulate smart kids who are going to be national leaders and god knows what else. You may not think you have much power now but you really do - go and get high positions, go and put pressure on these bastards to stop doing this. There’s a lot of stuff about aid but not that much publicity for debt relief. What about publicity about Britain selling arms? Fighting on those causes is something that people in our, rich countries have the legitimacy and standing to do because they’re citizens of those countries."

There is also some praise for DFID despite concerns about the strong incentives to keep dispersing no matter what:

"I was at DFID recently and they were actually much more receptive than I thought they would be. Many of these arguments are fully familiar to them, which tells me that it’s a very good aid agency. They’re people who don’t have their heads buried in the sand."

Read the rest here (and part one here), via Tom H

Statistical literacy at DFID

Some fascinating results from a new survey of DFID staff about their use and knowledge about evidence (credit to DFID for doing this and for publishing it), including these delicious stats:

84% seem to know what an RCT is
77% know what a census is, aaaaaaand
39% know what a national sample survey is.

hmmmmmmmmm........... maybe there is a point somewhere after all in all of this scepticism about the RCT hype ........?

Some of my best friends are knee-jerk leftists

I wrote a thing for the Guardian blog defending aid in support of private schools in developing countries. Which is very exciting because I've been reading the Guardian every day since I was 16. Some of the comments are a bit colourful, so for the record I feel I should burnish my lefty credentials (even though this feels really lame as it's exactly the kind of thing that annoys me when the likes of Goodhart and Collier do it before they go on to support mainstream Conservative party opinion).  

But for what it's worth, I started my lefty career when I was 6, when my "Dennis the Menace fan club says no Gulf war" poster made it to the local news. I went campaigning door-to-door for the Labour party when I was 8. I wrote to the Green party asking for a copy of their manifesto when I was 10. When I was 15 I vandalised the Conservative party billboard in my neighbourhood, and volunteered for a local Labour MP when I was 17.

I'm proud of having attended my local comprehensive school in Leeds. I'm proud that my fiercely liberal granddad sent my mum to the local comprehensive school on principle, instead of the more conveniently located selective school. I'm proud of my mum who was a school teacher for 20 years, and my aunt and uncle and grandparents, who all work or worked for the NHS (which yes, I'm also proud of).

I worry a lot about private schools in the UK, and the consequences for social mobility and segregation. 

So I'm not a natural supporter of private schools. But I care about evidence - and my reading of it is that there is a great potential to do good by experimenting with private sector service provision in education in developing countries. (Many other intelligent people, including several colleagues - none of whom are knee-jerk leftists - disagree with me, but thankfully none of them have yet accused me of "plain bullshit", "neo-con mantra", being a Mugabe-apologist, or a "twat.")

Where has all the education impact come from?

Lant Pritchett wrote an important paper back in 2000 called "Where has all the education gone." Despite a big increase in schooling in developing countries, there had been little increase on average in productivity.

Now I feel like we have the opposite problem - evidence that going to school has all sorts of wonderful impacts transforming lives

(- saves lives
- improves child nutrition
- increases job opportunities
- reduces child marriage
- reduces early birth
- makes people more tolerant
- leads to economic growth
- leads to more concern with the environment)

alongside evidence that many kids never actually learnt anything at school.

What's the deal? Are the positive impacts driven entirely by the kids who did learn something (the average treatment effect on both learners and non-learners), or is there something special and intangible (non-cognitive skills and character?) which kids can pick up from just sitting in a school even if they don't pick up any reading skills?

Any evidence?

Do iPhones increase gender inequality?

DFID have just published a new "topic guide" (quick evidence review?) on low cost private schools by Claire Mcloughlin of the University of Birmingham.

This isn't a criticism necessarily of Claire, but I am struck by how strange it is that the focus of the debate leads with how private schools affect equity. We are talking about countries such as Nigeria and Pakistan with net primary enrolment rates of less than 75%. I struggle to see how it could be a bad thing in such a context if some parents choose to spend their own money on private schools for their kids, even if no poor parents could afford it and all girls were totally excluded. Doesn't that just mean fewer kids for the state sector to fund? Of course in reality the data suggests that private schools in many countries have roughly similar gender access as public schools, and many poor people (though perhaps not the very poorest) also access private schools. 

Of course when we are talking about aid or government-funded places at private schools then equity should be a key concern, but for privately funded places, who cares? Isn't this totally missing the point? Do we worry about the equity implications of the new iPhone 5s for access to smartphones in the UK (actually I shouldn't be so hasty, soon being smart-phone-less probably will be the official definition of relative poverty). And when did I become so conservative?

Anti-social evidence-lite education advocacy

I'm trying to be a responsible sector specialist. As Owen Barder wrote way back in 2009.

we should, as a development community, heap scorn and opprobrium on anyone caught advocating for more resources in their sector. We need stronger social norms in development that frown upon this kind of anti-social behaviour.

We should be advocating for more resources for development, but these should be allocated across sectors by the best evidence not the best lobbying. We shouldn't be squabbling between ourselves over our pet projects. 

Sadly in the education sector not everyone seems to be on board with this message. Hugh Evans, the CEO of the Global Poverty Project, is fundraising for the Global Partnership for Education (GPE) this year.

Why? While achieving universal schooling by 2015 is a noble goal in and of itself, it must also be emphasised that investing in education is perhaps the most effective and quickest way to reduce poverty.

Uh oh... that sounds like a very confident statement. Does he have evidence for that? He has some:

Investing in education produces enormous yields. For instance, each additional year of schooling raises average annual gross domestic product growth by 0.37 per cent. Also, where the enrollment rate for secondary schooling is 10 per cent higher than the average rate for the population, the risk of war is reduced by around 3 per cent. And there is more and more evidence that proves increased access to education has significant flow on effects. Like the promotion of girls’ and women’s rights, falling infant mortality rates, and increased crop yields.

BUT. I'm pretty sure that all of that evidence he is referring to doesn't actually say anything about causality, and only really tells us something about correlations. There are no national-level experiments here. Countries that have higher rates of schooling may also have slightly faster growth, but we have known since the penis paper that looking at correlates of economic growth at the national level is mostly stupid. Countries with more schooling might indeed be less likely to be at war, but this DOES NOT prove that it was the schooling wot done it. The individual-level "micro" studies are generally more persuasive than the country-level "macro" studies, but even there most of them are looking at correlations rather than real or natural experiments.

Second - to make a statement that "education is perhaps the most effective and quickest way to reduce poverty" implies that you have also looked at the cost-effectiveness of all the other possible anti-poverty interventions. No discussion of that here.

And finally, no discussion at all of learning outcomes (see for example Schooling is not Education!). Keep up Hugh. 

Mathematising External Validity

Paul Krugman, Noah Smith, and Bryan Caplan had an interesting debate last week on the use (and misuse) of maths in economics.

A helpful illustration is provided this month by a new paper by Lant Pritchett and Justin Sandefur on external validity and RCTs (handy Charles Kenny summary in BusinessWeek here).

The concept of external validity is pretty simple to grasp intuitively. An experiment might give you a good estimate of the impact of a programme in a certain context, but it can't tell you if the same programme will have the same impact in a totally different context.

This is something which is especially obvious when you are actually working on national policy. When you are a writing a brief for a politician or an NGO on an issue, it would just feel stupid to lead with evidence from a totally different country if there is any data at all from the country you are actually in. Not that studies from other countries are uninteresting, but it is just blindingly clear that there are a lot of differences in political, social, and economic context between countries that might make results from a similar programme quite different, and so to use some caution in drawing conclusions, even from perfectly executed experimental studies.

At yet at the same time, "external validity" can seem like a bit of a hand-wavy rebuttal compared to all of the extensive technical theory around internal validity - whether your study is likely to give a biased estimate of programme impact on the population you are studying. There is a lot of detailed methodological analysis looking at exactly what the causes of bias are in different studies and how this bias can be best avoided. So what Pritchett's and Sandefur's paper does is add some detail to the our understanding of external validity, add some maths, and somehow make the critique seem somehow weightier. I think I still find their empirical examples more compelling than their theory, because I'm slow with maths and more interested in empirics, but nonetheless it does seem important to have that kind of systematic logical thinking through of the detail of a problem.

The bottom line:
- Economath - not totally useless, but you can probably get the intuition without it.
- External validity - an important concern, and sometimes contextual understanding matters more than clean identification - but also a reason for more experiments where possible not less

How you can literally give a whole family their entire annual income today for $1,000

Planet Money & This American Life have a really great new show on GiveDirectly.

What's best are the interviews with the recipients. It makes me feel very warm to imagine someone like Bernard Omondi in a remote village waking up one morning in slight disbelief to an SMS message containing $1,000 - his family's entire annual income - more cash than he has ever had at one time in his life - and that I sent him that.

A few things really struck me.

First, that even the recipients are sceptical about cash. Although they all explained how they themselves invested the money sensibly, they were quick to judge others in the village and explain how they had totally wasted theirs on booze. Perhaps this is a psychological thing and humans just have some kind of deep suspicion that everyone else is a bit of an idiot who needs taking care of? It reminds me of the surveys that show the overwhelming majority of drivers think that they are well above average (which is impossible). It's all the other road users who are idiots. Is that right? What is the relevant psychological bias? And what is the way of addressing this? The mounting evidence doesn't really seem to be doing the trick of convincing people.

Second, and related to that point, there seemed to be some surprise that many of the households had invested in a new metal roof. A chief criticism of someone who had "wasted" the transfer was "look at him, he still has a straw roof!" Now a metal roof which lasts for ten years actually turns out to be a great investment compared to a grass roof which needs constant costly maintenance. But I also wonder whether the fact that the transfer was explicitly targeted only on people with grass roofs had anything to do with people's choices? Might some people have got the new roof because they thought that's what the transfer was for?

Third, that "we are all poor around here." There are good reasons to think that selective-targeting of transfers can damage social relations. One of the recipients says "I've lost a few friends." Which could be important if you then later need to rely on those friends when you're in a tough spot. And so GiveDirectly are trying out a new model, identify a poor village and then give cash to everyone in the village, something that people asked for repeatedly when I spoke to recipients in Kenya and Eastern DRC.

Finally, there is an interview with a woman from Heifer International, which Chris Blattman comments on here (I think he actually goes a bit lightly on them). I actually think that sending a cow could be a great intervention, but it was just an embarrassing performance, which suggested that the organisation doesn't seem to believe at all that data can be informative about effectiveness, and doesn't seem to care at all about value for money (Heifer can't or won't even provide a per family cost for their programme, because "every family is different"). My money will keep going to the organisations interested in evidence and value for money. 

Evidence-based policy-making US-style

Based on our rough calculations, less than $1 out of every $100 of government spending is backed by even the most basic evidence that the money is being spent wisely.

...

Since 1990, the federal government has put 11 large social programs, collectively costing taxpayers more than $10 billion a year, through randomized controlled trials, the gold standard of evaluation. Ten out of the 11—including Upward Bound and Job Corps—showed “weak or no positive effects”

Just in case you thought that there was any danger of the whole results agenda and RCT-fetishism taking over in American politics. From an excellent piece in last month's The Atlantic, which by the way is generally fantastic, I just bought a paper copy for the first time and the whole thing was full of interesting - a critical look at the evidence on over 35 female fertility, a thing about how much health food is actually really unhealthy, a note about how recycling can actually increase carbon emissions because it needs more trucks on the streets, and a piece discussing relationships and gender politics and family from the perspective of a man who has sacrificed his career for his wife's. 

More Disappointing Labour Market Policy Outcomes

This new from Jordan:

Wage subsidies and soft skills training are two popular types of policies that governments are turning to around the world as part of their efforts to deal with high youth unemployment. Our experimental analysis shows these policies do not appear to have had large impacts on generating sustained employment for young, relatively educated women in Jordan. Short-term wage subsidies generated large and significant increases in employment while the subsidies were in effect, but most of these jobs disappeared when the subsidies expired. High minimum wages may be one reason, with firms saying that graduates were not productive enough to be affordable without subsidies.

Groh, Krishnan, McKenzie, and Vishwanath, The impact of training and wage subsidy programs on female youth employment in Jordan

I don't see much cause for optimism in getting any solid positive results from labour market interventions. Am I missing something?

Chart of the day: Evidence-based aid in the UK

This chart from the LSE "Impact of Social Science" handbook shows a ranking of UK government departments by the number of references to academic research found on their websites. DFID comes third.

Does slum upgrading work?

A new colleague at OPM Ruhi Saith is a co-author of a new Cochrane systematic review on the impact of slum upgrading programmes on health and wellbeing (full summary here).

They find only 5 studies which can demonstrate any causality, from which they find:

• "Limited but consistent evidence to suggest that slum upgrading may reduce diarrhoea in slum dwellers and that slum dweller’s water related expenses may also be reduced
• Mixed results for whether slum upgrading can reduce parasitic infections, educational outcomes, financial poverty and unemployment outcomes
• Very little information on other health or social outcomes, or which types of interventions were most beneficial"

Which reminds me of two things,

first, John Snow and the 1854 Broad Street cholera epidemic, when John used a mixed methods approach based on KII*, and a pathbreaking geographic data visualization infographic** which founded the science of epidemiology using one of the first natural experiments.

second, that there is really weak evidence that area-based initiatives have any impact on employment and well-being in the UK, and so policy should target people not places. 

Which suggests that slum upgrading should focus on providing the public goods and infrastructure with clear evidence of impact and cost effectiveness - namely clean water and sanitation - and be more modest about expectations for impacts on other outcomes which are not primarily determined by the slum environment, such as poverty, unemployment, and low education.

*Key informant interviews. Or talking to people. Yes I am mocking your terminology, quals.

**A map. Yes you too, data monkeys.

Bigot of the Year

Britain's most senior Catholic has been named "bigot of the year" by Stonewall for writing that gay relationships are "harmful to the physical, mental and spiritual wellbeing of those involved," that gay marriage is "madness" and a "grotesque subversion" of a human right, and making a bizarre analogy between the introduction of gay marriage and the reintroduction of slavery.

All of which is pretty disgusting, but I'm such a nerd that I'm almost more annoyed when he explicitly uses the word "evidence" when I'm pretty sure there is no such evidence.

All children deserve to begin life with a mother and father; the evidence in favour of the stability and well being which this provides is overwhelming and unequivocal. It cannot be provided by a same-sex couple, however well-intentioned they may be.

Cardinal, that's a step too far. It's also personal - I managed without a father just fine thanks for your concern Cardinal. All children deserve love but the gender of their parents is irrelevant, and that is an evidence-based statement.

Research has shown that the kids of same-sex couples — both adopted and biological kids — fare no worse than the kids of straight couples on mental health, social functioning, school performance and a variety of other life-success measures. 

In a 2010 review of virtually every study on gay parenting, New York University sociologist Judith Stacey and University of Southern California sociologist Tim Biblarz found no differences [my emphasis] between children raised in homes with two heterosexual parents and children raised with lesbian parents.

Is there an award for evidence-abuser of the year?

How do we scale up personalized support? (policy response to heterogeneity)

The CGAP Graduation pilots seem to be getting good results (evaluations with Esther Duflo here and Dean Karlan here). My instinctive reaction is that personalised, individualised, tailored support may very well be successful, but is it possible to set up systems to implement this routinely at scale?

The need seems to be clear. Perhaps the key bottom line from the microfinance impact literature is heterogeneity - people are different, their needs are different, and one-size-fits-all policy has all sorts of different impacts, both positive and negative, on different kinds of people.

Similar results seem to be emerging on support for small businesses (a lit review by David McKenzie here and and evaluation from Ghana here), and in education and early child development (see this great episode of This American Life covering Paul Tough's new book How Children Succeed).

Meanwhile in the UK, Iain Duncan Smith has decided that the benefits system is far too complicated (it is complicated) and so it needs to be simplified, rolling 6 different benefits into one "Universal Credit." But maybe, just maybe, complicated people need complicated support? And is that a realistic goal for developing countries with weak government systems?

Does deworming really work?

The latest Cochrane Collaboration review of the evidence on the impact of deworming on various outcomes has come out decidedly less than optimistic.

Here's a summary by the very smart Alexander Berger from Givewell, some discussion on the Public Library of Science blog including comment from one of the Cochrane authors and Alan Fenwick from SCI, and finally a rebuttal to the review's findings on schooling from IPA, JPAL, CEGA, Deworm the World, and the authors of the original Busia experiment on deworming.

I haven't spent enough time looking at the details to come to a strong opinion here, but one point made on the IPA blog seems evidently correct - random assignment should be enough to ensure pre-treatment balance between treatment and control. That is the whole point of random assignment. And  following the recent debacle of the medical journal the Lancet being forced to retract the key finding of a social-science-y study after some actual social scientists pointed out a mathematical error, combined with my disciplinary and professional loyalties, I'm inclined to go with the social scientists rather than the doctors on this one.


Update: Thoughts from David McKenzie

The Impact of IPA

Measuring policy influence is hard, but this looks like a slam dunk for IPA: Concern Worldwide are launching a new Gates Foundation funded initiative focused on Innovation, Evidence, Action, and What Works.

Compare and contrast the Concern website;

and the IPA website;

Congrats all round.