Can we randomize development evaluation?
Jon Lomoy, Director of the OECD Development Co-operation Directorate (DCD-DAC) presents the first of two contributions to the debate on the use of randomised control trials in development economics. Diana Coyle will reply.
In her post Is economics leaving Wonderland?, Diane Coyle justly notes that development economics is ahead of the curve when it comes to “shedding the belief, at least on the part of many economists, that a single conceptual approach will deliver a ‘silver bullet’ solution or method that can be applied everywhere.”
Yet I still think we need to ask whether academic innovations in the multidisciplinary approach to economics, particularly development economics, have had an on-the-ground impact on the practice of development co-operation.
There has been a lot of discussion in academic circles, for example, about the use of randomised control trials to improve the outcome of development co-operation. While I welcome the emphasis this implies on results – and this is why evaluation has such a valuable role to play in development – like many, I wonder about the approach.
Randomised control trials in medicine make sense to me. To find out if a medication works well, you give it to one group of patients and a placebo to another. Then you see if the group taking the medication improves significantly over the group taking the placebo. Simple.
But can we take it from there to assume, as does the OECD colleague Diana Coyle quotes, that in a few years’ time randomised control trials will “turn out to have transformed the field” of development economics? Naturally, we want development that works – we want to see obvious, visible progress in poverty reduction and development. We want to validate the investment of time, money, and mental and physical energy. And where success is elusive, we want to understand quickly why we failed and learn how to do things better.
Yet the narrow, medical approach of randomised trials can, I believe, tell us very little when we need to provide information that is useful to decision makers. When it comes to understanding whether higher-level policies and programmes are working, or whether micro-successes are adding up to real progress, we need a different kind of evaluation. For example, randomised control trials can tell us whether or not an individual project for reintegrating ex-combatants worked well, but will give us little insight into the overall politics and dynamics of peacebuilding – and into how international partners can best help in settings of violent conflict. Will they help donors decide whether they can best support education by channelling money through national budgets or by financing local NGOs? Furthermore, given what we know about the diversity of development contexts, problems like this are multiplied when we try replicate positive experiences in different countries.
Evaluating development impact is not simple. The simple approach of randomised control trials can help to treat specific symptoms, but may leave larger questions of how to end “the ailment” unanswered.
Dr Coyle does well to point out “the crucial importance of the specific context.” I have seen providers of development assistance struggle to understand how their support makes a difference. How do they use findings from treating mosquito nets with insecticide to lower the rate of malaria worldwide, or from using mobile phones to improve the livelihoods of rural farmers to generate knowledge that will make development co-operation more effective at the global policy level? It’s not easy – especially when thinking about ‘global public goods’ like the environment, education, gender equality, or fair trade. With so many factors to consider, a host of different evaluation methods are needed to tackle them.
So while I very much endorse Dr Coyle’s hope that development economics has left Wonderland, I also hope that academic theory is heading for the real world – aiming to turn its incredible range of knowledge into practical insights and tools that can be used to treat the ailment rather than the just the symptoms.
There is, indeed, no “silver bullet” to cure a global problem like poverty. It is great that we recognise this – as long as it doesn’t discourage us in our quest to end poverty. And it is great that we celebrate leaving Wonderland – as long as we realize that understanding all the pieces of the puzzle, and how they fit together to guide effective action, requires a much broader approach to evaluation.