Mathematising External Validity

Paul Krugman, Noah Smith, and Bryan Caplan had an interesting debate last week on the use (and misuse) of maths in economics.

A helpful illustration is provided this month by a new paper by Lant Pritchett and Justin Sandefur on external validity and RCTs (handy Charles Kenny summary in BusinessWeek here).

The concept of external validity is pretty simple to grasp intuitively. An experiment might give you a good estimate of the impact of a programme in a certain context, but it can't tell you if the same programme will have the same impact in a totally different context.

This is something which is especially obvious when you are actually working on national policy. When you are a writing a brief for a politician or an NGO on an issue, it would just feel stupid to lead with evidence from a totally different country if there is any data at all from the country you are actually in. Not that studies from other countries are uninteresting, but it is just blindingly clear that there are a lot of differences in political, social, and economic context between countries that might make results from a similar programme quite different, and so to use some caution in drawing conclusions, even from perfectly executed experimental studies.

At yet at the same time, "external validity" can seem like a bit of a hand-wavy rebuttal compared to all of the extensive technical theory around internal validity - whether your study is likely to give a biased estimate of programme impact on the population you are studying. There is a lot of detailed methodological analysis looking at exactly what the causes of bias are in different studies and how this bias can be best avoided. So what Pritchett's and Sandefur's paper does is add some detail to the our understanding of external validity, add some maths, and somehow make the critique seem somehow weightier. I think I still find their empirical examples more compelling than their theory, because I'm slow with maths and more interested in empirics, but nonetheless it does seem important to have that kind of systematic logical thinking through of the detail of a problem.

The bottom line:
- Economath - not totally useless, but you can probably get the intuition without it.
- External validity - an important concern, and sometimes contextual understanding matters more than clean identification - but also a reason for more experiments where possible not less

DFID Livelihoods Program in South Sudan

DFID is planning to spend up to £100 million on food security and livelihoods in South Sudan over the next 5 years, the largest of all its programs. Is that a lot or a little?

DFID's expected results in this area are to support 1 million people to achieve food security.

Not knowing the details of the program, I am going to imagine for a second that DFID has a zero-overhead cash transfer or food voucher planned. 

£100 million over 5 years, divided between £1 million people, is 5.4 pence a day each. 

"Hello there Mr. Deng, here's 5 pence, buy yourself a sandwich yeah? Go nuts with it, I'll give you another 5p tomorrow! Sorted yeah?"

So - to get to an even slightly more realistic sufficient basic daily income, all we need is for economies of scale, support to production, and that vocational training, to have a 1000% return. Good job that we have all that evidence about the massive massive returns to livelihoods programs. Wait...

The quantity fallacy of ineffective aid

Within an interesting article on returnee Sudanese diaspora, I spotted this nugget which falls victim to the John Quiggin quantity fallacy of ineffective aid:

Since the peace agreement, donor countries have pumped more than $1 billion in aid into South Sudan—to little noticeable effect outside the capital. This is a place where the nicest homes and office buildings are little more than prefabricated containers—most people live in mud huts with conical straw roofs or in single-story cinder-block buildings, and glass windows are still a luxury.

Wow a BILLION dollars! That's a lot right? Or is it? To paraphrase Penn is a billion a shedload or a motherfunking shedload? It basically may as well be quadrillion for all my little head can cope.

So how about we scale it down to something understandable. Not that the aid was handed out to actual people, but imagine for a second it was.

$1 billion divided by 8 million Southern Sudanese people = $125 each.

Over 5 years that is $25 a year each.

So, Mr. Deng, we've been giving you $25 a year for FIVE YEARS now and you still don't have anything to show for it. Clearly then aid doesn't work or it has all been wasted or stolen.