14 May 2012

Millennium Villages: impact evaluation is almost besides the point

A lot has been said about evaluation and the impact of the Millennium Villages, most of which boils down to:

"What is the impact of the Millennium Village package of interventions on the area in question?"

The really depressing part though is that this is actually the least interesting question. Chances are that throwing in a whole bunch of extra inputs to a community will create some outputs, and some impact. The whole point of the Millennium Villages though is to provide a model for the rest of rural Africa to follow. The really interesting question is whether African governments have the desire and capability to manage a massive and complex scaling up of integrated service delivery across rural Africa.

A point which basically belongs to Bill Easterly.
Mr. Easterly argues that the Millennium approach would not work on a bigger scale because if expanded, “it immediately runs into the problems we’ve all been talking about: corruption, bad leadership, ethnic politics.” 
He said, “Sachs is essentially trying to create an island of success in a sea of failure, and maybe he’s done that, but it doesn’t address the sea of failure.” 
Mr. Easterly and others have criticized Mr. Sachs as not paying enough attention to bigger-picture issues like governance and corruption, which have stymied some of the best-intentioned and best-financed aid projects.
A proper randomised evaluation could give you a good estimate of the cost-effectiveness of the island. A difference-in-difference estimate could give you a slightly worse estimate. Doing a fake difference-in-difference with unreliable recall baselines, arbitrarily selected control villages, misrepresented results, and mathematical errors, will give you a pretty awful estimate. But either way, you are missing the main point, which is about scale and replication, and how that works.

How feasible would it really be to replicate something like this on a national level in Ghana? How exactly would it work? Do the  systems of accountability and capability exist at local levels to manage all of these projects? How would coordination and planning work between national ministries and their sectoral plans, and local level priorities?

The Millennium Village project seems to grasp vaguely at these issues but ultimately brush them under the table. From a MV project report:
Another challenge in some sites is insufficient capacity of local government to take full ownership of MV activities. This is manifested in unfulfilled pledges to perform mandated roles, unsatisfactory maintenance of infrastructure, and insufficient involvement of local elected officials. MV site teams are addressing these challenges by agreeing to jointly implement interventions targeted at improving the performance of sub‐district governments, increasing sensitization and engagement of local government officials, increasing joint monitoring of MV activities in communities, and developing training plans in technical, managerial, and planning skills for local government officials.
 Or : "we have no clue how to fix the systemic implementation challenges"

An anonymous aidworker writes on his blog Bottom-up thinking
I’ve noticed around here, normally sloth-like civil servants who won’t even sit in a meeting without a generous per diem rush around like lauded socialist workers striving manly (or womanly) in the name of their country when a bigwig is due to visit, working into the night and through weekends, all without any per diems...   
I fear all the achievements of the MVP will wash up against the great brick wall that is a change resistant bureaucracy.
None of this is to say that the situation is hopeless. It isn't. In particular there are elements of the Millennium Village package which are proven to be effective, cheap, and don't require complicated systems of governance and accountability. Namely distributing insecticide-treated bednets. Aid money can provide them easily, sustainability is less of a concern than other interventions, and you can buy them right now. Check out Givewell for a rigorous independent assessment (and recommendation) of the Against Malaria Foundation. Probably the single best way you could spend some money today. 


Post a Comment