Mathematising External Validity

Paul Krugman, Noah Smith, and Bryan Caplan had an interesting debate last week on the use (and misuse) of maths in economics.

A helpful illustration is provided this month by a new paper by Lant Pritchett and Justin Sandefur on external validity and RCTs (handy Charles Kenny summary in BusinessWeek here).

The concept of external validity is pretty simple to grasp intuitively. An experiment might give you a good estimate of the impact of a programme in a certain context, but it can't tell you if the same programme will have the same impact in a totally different context.

This is something which is especially obvious when you are actually working on national policy. When you are a writing a brief for a politician or an NGO on an issue, it would just feel stupid to lead with evidence from a totally different country if there is any data at all from the country you are actually in. Not that studies from other countries are uninteresting, but it is just blindingly clear that there are a lot of differences in political, social, and economic context between countries that might make results from a similar programme quite different, and so to use some caution in drawing conclusions, even from perfectly executed experimental studies.

At yet at the same time, "external validity" can seem like a bit of a hand-wavy rebuttal compared to all of the extensive technical theory around internal validity - whether your study is likely to give a biased estimate of programme impact on the population you are studying. There is a lot of detailed methodological analysis looking at exactly what the causes of bias are in different studies and how this bias can be best avoided. So what Pritchett's and Sandefur's paper does is add some detail to the our understanding of external validity, add some maths, and somehow make the critique seem somehow weightier. I think I still find their empirical examples more compelling than their theory, because I'm slow with maths and more interested in empirics, but nonetheless it does seem important to have that kind of systematic logical thinking through of the detail of a problem.

The bottom line:
- Economath - not totally useless, but you can probably get the intuition without it.
- External validity - an important concern, and sometimes contextual understanding matters more than clean identification - but also a reason for more experiments where possible not less

When rigorous evaluation is NOT necessary?

Sometimes simple descriptive survey data is basically fine to tell us what we need to know. But if you want to get published in a top economics journal, you need a really convincing statistical demonstration of causality (unlike, say, some top medical journals *cough* American Journal of Clinical Nutrition *cough*).

Case #1 - Oster & Thornton's very cool paper on menstruation and school attendance. They found two things
(a) - from a simple survey/school attendance records - girls in Nepal miss an average of 0.4 days per 180 day school year due to menstruation.

(b) - a randomized intervention providing sanitary products has little impact on reducing that 0.4 days.

Now - the interesting part here is part a - based on simple descriptive statistics - which tells us everything we need to know. Part b - the randomized intervention - is basically irrelevant once we already know that girls are not missing school due to menstruation

Case #2 - Friedman, Kremer, Miguel & Thornton - on education and attitudes to democracy and ethnicity. By offering randomized primary school scholarships they get a very clean identification of the causal impact of education on attitudes. Which is great. But we also knew that there is a ton of descriptive statistics on this already.

From this simple cross-tabulation of Afrobarometer data (the website is pretty cool by the way) - I'm guessing that primary school probably doesn't have a huge impact on support for democracy, because almost everyone supports it to begin with. 

Everything you ever wanted to know about RCTs and Microfinance but were too afraid to ask

If you’re looking for a thorough, careful, and clear summary of the latest research, and one that gets to those important design questions, you can do no better than read this review. My own chapter 6 is not nearly as complete and precise. --- David Roodman

So, You Want to Be a Scientist?

Is apparently a real thing. My mum just turned on BBC Radio 4 to a set of listeners calling in with questions that they would like to try and answer with an experiment - does windy weather make kids act up at school - do cows really lie down just before it starts to rain - and a panel discussing the practicalities of possible experimental design.

From the website, here is Brian Cox on what makes a good experiment;

"experiments are the most important part of science"