Exciting new results from the Millennium Village in Ghana

Only joking. Six years in, and Bonsaaso [the MV in Ghana] primary school has no toilet.

"Our toilet facilities at Bonsaaso primary school are not working at all," said Owusu [acting headteacher]. "The children have to go home when they need to use the toilet."

The Lancet's editors don't get evaluation (sadface)

So Matt beat me to the punch on Friday on the Lancet Millennium Village retraction. Since then I've being trying to think of a polite way of expressing my total dismay and despair at the tripe written by the Lancet editors in response to the retraction (for which, by the way, a little bit of Kudos to Pronyk et al).

The Lancet editors write:

The Millennium Villages project team has quickly and commendably corrected the record after understanding the validity of the challenge it received. But the withdrawal of this element of the paper does not detract from the larger result—namely, that after 3 years Millennium Villages saw falls in poverty, food insecurity, stunting, and malaria parasitaemia, together with increases in access to safe water and sanitation.

Which is just total nonsense. For all we know, poverty fell in the Villages at the exact same rate as everywhere else. That is not an important result to be celebrated. I challenged Lancet editor Richard Horton on twitter as to why he would continue to emphasise this non-result, and he responded with yet more nonsense;

1. richardhorton1 @rovingbandit To be fair, there were falls in each of the 5 MDG-1 poverty/nutrition measures, but these were not statistically significant. 18 May 2025 from web

That isn't even true. The first of the measures - wealth - is the opposite of poverty. It is *wealth* that fell (statistically insignificantly) in the Villages relative to comparisons. I despair. And kind of question my own sanity. Despite what Tim Worstall says, I'm really not a scientist, but its pretty galling that people say economics is not a science like the physical sciences when this is the kind of guff published by the world's top medical journal.

Bill Easterly has a whole long list here of more terrible social science published in medical journals. At the bottom of the post, Ben Goldacre comments

i think journals publishing things outside of their field of expertise is risky, but i wld caution against developing a world view that economics journals are in a better shape overall than medical ones. as someone who flits into both, there are lots of things that are routine in medical journals, to a greater or lesser extent, but notably almost unheard of in economics. stuff like declarations of conflict of interest, structured write-ups, registering a protocol in advance of doing a study, etc. all of which wld be great to see more of outside medicine.

All of which is true. In particular I am struck by how easily readable a short, structured, 4 page Lancet write-up is. There are definitely lessons to be learnt across disciplines both ways. It's just an incredibly sad state of affairs that one of the lessons that journals of medicine, the discipline that gave us randomized controlled trials, needs to learn from economics, is a more careful attention to statistics and causality. 

Millennium Villages: impact evaluation is almost besides the point

A lot has been said about evaluation and the impact of the Millennium Villages, most of which boils down to:

"What is the impact of the Millennium Village package of interventions on the area in question?"

The really depressing part though is that this is actually the least interesting question. Chances are that throwing in a whole bunch of extra inputs to a community will create some outputs, and some impact. The whole point of the Millennium Villages though is to provide a model for the rest of rural Africa to follow. The really interesting question is whether African governments have the desire and capability to manage a massive and complex scaling up of integrated service delivery across rural Africa.

A point which basically belongs to Bill Easterly.

Mr. Easterly argues that the Millennium approach would not work on a bigger scale because if expanded, “it immediately runs into the problems we’ve all been talking about: corruption, bad leadership, ethnic politics.” 

He said, “Sachs is essentially trying to create an island of success in a sea of failure, and maybe he’s done that, but it doesn’t address the sea of failure.” 

Mr. Easterly and others have criticized Mr. Sachs as not paying enough attention to bigger-picture issues like governance and corruption, which have stymied some of the best-intentioned and best-financed aid projects.

A proper randomised evaluation could give you a good estimate of the cost-effectiveness of the island. A difference-in-difference estimate could give you a slightly worse estimate. Doing a fake difference-in-difference with unreliable recall baselines, arbitrarily selected control villages, misrepresented results, and mathematical errors, will give you a pretty awful estimate. But either way, you are missing the main point, which is about scale and replication, and how that works.

How feasible would it really be to replicate something like this on a national level in Ghana? How exactly would it work? Do the  systems of accountability and capability exist at local levels to manage all of these projects? How would coordination and planning work between national ministries and their sectoral plans, and local level priorities?

The Millennium Village project seems to grasp vaguely at these issues but ultimately brush them under the table. From a MV project report:

Another challenge in some sites is insufficient capacity of local government to take full ownership of MV activities. This is manifested in unfulfilled pledges to perform mandated roles, unsatisfactory maintenance of infrastructure, and insufficient involvement of local elected officials. MV site teams are addressing these challenges by agreeing to jointly implement interventions targeted at improving the performance of sub‐district governments, increasing sensitization and engagement of local government officials, increasing joint monitoring of MV activities in communities, and developing training plans in technical, managerial, and planning skills for local government officials.

 Or : "we have no clue how to fix the systemic implementation challenges"

An anonymous aidworker writes on his blog Bottom-up thinking

I’ve noticed around here, normally sloth-like civil servants who won’t even sit in a meeting without a generous per diem rush around like lauded socialist workers striving manly (or womanly) in the name of their country when a bigwig is due to visit, working into the night and through weekends, all without any per diems...   

I fear all the achievements of the MVP will wash up against the great brick wall that is a change resistant bureaucracy.

None of this is to say that the situation is hopeless. It isn't. In particular there are elements of the Millennium Village package which are proven to be effective, cheap, and don't require complicated systems of governance and accountability. Namely distributing insecticide-treated bednets. Aid money can provide them easily, sustainability is less of a concern than other interventions, and you can buy them right now. Check out Givewell for a rigorous independent assessment (and recommendation) of the Against Malaria Foundation. Probably the single best way you could spend some money today. 

OMG Millennium Villages Increase Poverty ROFL!!

The Millennium Village PR Department Guardian newspaper reports "Child mortality down by a third in Jeffrey Sachs's Millennium Villages." Which is possibly true (I'm not going to even go into the validity of the non-random controls). But if you take a casual glance at the paper's results table, you'll also find no statistically significant impact of the project on poverty, nutrition, education, or child health.

Of all 18 indicators, 10 are totally statistically insignificant (no difference between intervention and comparison) and only 1 of the 18 indicators is significant at the 1% level.

The text of the Lancet paper mentions 3 times that poverty has fallen in the village sites. And just once that this reduction is actually no different to that in comparison villages.

And check out this sentence;

For 14 of 18 outcomes, changes occurred in the predicted direction. No significant differences were recorded when comparing poverty ...

So, mention the direction of the effect when it is the direction you want (but statistically insignificant from zero), and neglect to mention the direction of the effect when it is the direct opposite of what you want (but also insignificant).

Now THAT, folks, is science. (Here's the Lancet link, HT: Maham). 

Easterly on Millenium Villages

“Sachs is essentially trying to create an island of success in a sea of failure, and maybe he’s done that, but it doesn’t address the sea of failure.”

via Derrill Watson