Follow
FSG
Discovering Better Ways to Solve Social Problems

Fix That Fits: What is the Right Evaluation for Social Innovation?

Does microcredit reduce poverty? Do laptops in schools improve student learning? If you believe in Randomized Control Trials (RCTs) as the only way to evaluate an intervention, the answers to the questions are: No and No.

In 2009, researchers from the Poverty Action Lab at the Massachusetts Institute of Technology (MIT) worked with an Indian microfinance firm to ensure that 52 randomly chosen slums in the city of Hyderabad were given access to microfinance, while 52 other slums, which were equally suitable and where the lender was also keen to expand, were denied it. The study found that there was no effect on average household consumption (a proxy for income), at least within 12-to-18 months of the experiment.

Earlier this year, a group of researchers from the Inter-American Development Bank (IDB) conducted a randomised evaluation of the One Laptop Per Child (OLPC) programme, using data collected after 15 months of implementation in 319 primary schools in rural Peru. They found that the children receiving laptop computers under the OLPC programme did not show any improvement in mathematics or reading.

Seems like a straightforward vindication of the power of RCTs to uphold the truth and prevent wastage of social sector resources? Unfortunately, the reality is a bit more complicated.

In the microcredit study for instance, although overall consumption did not go up, people in the slums of Hyderabad who had access to microcredit were more likely to cut down on tobacco and alcohol in favour of more durable goods such as pushcarts or cooking pans (that helped further their business).

Consequently, the MIT researchers discovered  that as many as one-third more businesses had opened in slums which had a microcredit branch, leading one to believe that while there was no immediate effect, there would certainly be an impact on poverty (and health) in the long run.

Similarly, in the Peru study, the IDB researchers noted that while there wasn’t a measurable improvement in test scores, there was a positive and significant change in the development of children’s cognitive skills, which is often a more highly valued outcome.

More importantly, the study’s findings shed light on the challenging context of Peru’s education system, specifically around the lack of adequate resources and poor teacher-training. For instance, 62 percent of Peruvian teachers did not have elementary level reading comprehension, and 92 percent lacked acceptable proficiency in mathematics.

In both these cases, just following the headline finding would have led to the demise of two potentially promising interventions. Increasingly, we are coming up against the limitations of conventional evaluation methods being applied to new and innovative approaches.

Giving slum-dwellers access to microcredit and rural children access to laptops are very different programmes, but they have something in common: They are both “social innovations”. In other words, they are novel solutions to a social problem that are potentially more effective, efficient, sustainable, or just, than present solutions.

Social innovations often seek to address problems that are both complicated (with many moving parts) and complex (including and interdependence of variables, multiple factors interacting at once, iterative and non-linear feedback loops, and rapid change in dynamic contexts). They often share the following traits:

  • The pathways to results and sometimes even the results themselves are unpredictable and emergent, such as the result around slum-dwellers using more of their income for durable goods.
  • When many different independent individuals, organizations, and institutions affect a problem and its solution, it can be difficult to produce specific outcomes at a pre-determined time (e.g. test scores at the end of the year).
  • Social innovators do not have enough control over the entire scope of factors or players to orchestrate outcomes. Often, the context may be the critical factor (as with the poorly prepared teachers of Peru) rather than the intervention itself.

 

In short, people who test new solutions to complex problems do not have the luxury of a clear or proven set of activities and processes. Hence, summative (Did the programme work?) studies such as RCTs, are often ill-suited to evaluating social innovations. Even a formative evaluation (Is the programme making progress?) assumes that a programme can clearly specify causal pathways that will lead to predictable outcomes, which is not always possible with social innovations where there is significant uncertainty about how the intervention will unfold.

It is quite likely that using a formative or summative approach may not only result in inaccurate or meaningless information, but it may also squelch the adaptation and creativity that is so vital to social innovation.

What IS the fix that fits?

An emerging approach called Developmental Evaluation (DE) is gaining traction among some fund-givers who support collaborative, complex, evolving change processes. Originally conceptualised and described by evaluator Michael Quinn Patton, Developmental Evaluation “Informs and supports innovative and adaptive development in complex dynamic environments. DE brings to innovation and adaptation the processes of asking evaluative questions, applying evaluation logic, and gathering and reporting evaluative data to support project, programme, product, and/or organisational development with timely feedback.”

In FSG’s recently released white paper, Evaluating Social Innovation , the authors suggest that developmental evaluation has five characteristics that distinguish it from formative and summative evaluation approaches.

These include the focus of the evaluation, the intention of learning throughout the evaluation, the emergent and responsive nature of the evaluation design, the role and position of the evaluator, and the emphasis on using a systems lens for collecting and analysing data, as well as for generating insights. As such, a developmental evaluation focuses on the following questions:

  • What is developing or emerging as the innovation takes shape?
  • What variations in effects are we seeing?
  • What do the initial results reveal about expected progress?
  • What seems to be working and not working?
  • How is the larger system or environment responding to the innovation?
  • How should the innovation be adapted in response to changing circumstances?
  • How can the project adapt to the context in ways that are within the project’s control?

 

Several conditions such as organisational fit and readiness need to be in place to ensure a successful developmental evaluation effort. However, when implemented well, DE provides both real-time data that informs programmatic and strategic decision-making processes, and an opportunity to engage in “sense-making” processes that help shape social innovations in ways that help them achieve their goals and long-term outcomes.

How does DE apply in the Indian context?

The philanthropic and social sector in India has grown significantly in the past decade. Social entrepreneurs have created ways to improve society that build on the culture of innovation and adaptation that is unique to India. From delivering high quality eye surgery at a fraction of the West’s price to creating the $2,000 family car, the philosophy of jugaad runs through Indian society.

However, as a recent report from the Dasra Foundation and the Omidyar Network notes, “Impact Assessment is shrouded in ambiguity and jargon, with little to no convergence in thought on the topic.” It would be a mistake for the Indian social sector to embrace conventional evaluation approaches and methods that do not fully account for the complexity and dynamism of Indian society.

In order to fully realise the power of social innovation in India, a new approach to evaluation is needed. Developmental evaluation serves that need by maintaining the rigour of decision-making based on evidences, while allowing for a more real-world approach that is cognizant of cultural and contextual factors.

By Hallie Preskill, FSG Managing Director, and Srik Gopalakrishnan, FSG Director
Hallie oversees the firm’s Strategic Learning & Evaluation approach area. Prior to joining FSG, Hallie spent more than 20 years in academia, teaching graduate level courses in program evaluation and organizational learning, among others. She has authored several books and articles on evaluation related topics, and served as the President of the American Evaluation Association in 2007.  Srik has worked in the public and private sectors and has extensive experience in program evaluation, measurement and learning. He has led developmental, formative and summative evaluations, as well as projects on organizational design, strategic planning and leadership development. 

Post Your Comment
Name
Required
Email
Required, will not be published
Comment
All comments are moderated
 

Comment

Pingback: My Homepage

OLPC Peru is certainly not the best example for the points the author tries to make. OLPC Peru is a very sad case of poor people whose very limited resources have been wasted (due to government corruption) on a pseudo-scientist's demented education dreams ("we give a kid a cheap laptop, he won't need a teacher"). Negroponte and the Peruvian government have hurt those who need the most protection.
Truth In Advertising - Yes, Advertising After reading this article - which appears rather a PR effort to sell a consulting product than a real contribution to important questions - the mind boggles about the added value of the "developmental evaluation" which is advertised here. While much is wrong with randomised controlled trials (RCTs), particularly their implicit assumption that the alternative to the intervention is to do nothing at all, at least they search for impact. On the other hand, the advertised evaluation technique obfuscate sthe question of impact with such questions as "What do the initial results reveal about expected progress?" or "What seems to be working and not working?", focusing instead on how to make the project itself survive ("How should the innovation be adapted in response to changing circumstances?; How can the project adapt to the context in ways that are within the project’s control?"). But what is the intrinsic value of a project whose impact cannot be shown? It seems that the advertised technique is more concerned with the welfare of the project than of its intended beneficiaries. The problems experienced by the authors in understanding results of RCTs may partly explain their aversion to RCTs. Their statements about the Hyderabad RCT are either misinformed or simply misleading. They cite as hidden successes of microfinance that "although overall consumption did not go up, people in the slums of Hyderabad who had access to microcredit were more likely to cut down on tobacco and alcohol in favour of more durable goods such as pushcarts or cooking pans (that helped further their business)." First, what Preskill & Gopalakrishnan fail to truthfully mention is that these same people also cut back on TEA and FOOD consumption; cutting back on essentials of life which actually suggests that microdebt made these neighbourhoods worse-off. Second, the reported higher investment in durable goods was 12 Rupees in the microfinance neighbourhoods versus 5 Rupees in others; a difference of nine cents per month, or roughly the price of a samosa, which is unlikely to have any short-run or long-run impact. Further, they note that "the MIT researchers discovered that as many as one-third more businesses had opened in slums which had a microcredit branch, leading one to believe that while there was no immediate effect, there would certainly be an impact on poverty (and health) in the long run." ; another very selective statistic, and one which should hardly "lead one to believe" in long impact (note the wish to "believe" here). The RCT researchers in fact reported (on p. 17) that "5.3% of households opened a business in the year prior to the survey, compared to 7% in treated areas", and only "1 in 5 of the additional MFI loans in treatment areas is associated with the opening of a new business". So microloans hardly did anything at all. If the actual aim is to measure long-term successes, rather than "believe", the question which matters is how many of these businesses actually survive their first two years of existence. And for that, one needs impact evaluation; not the wishy-washy consulting advertised here.
Thank you for your comments. We would be happy to respond to the points that you have raised: 1) Truth in advertising: Developmental Evaluation is a legitimate field within the broader evaluation field, with several consultants and practitioners actively involved in DE efforts. DE is especially garnering interest in the social sector where new and innovative approaches to solving social problems are being tested. The recent American Evaluation Association conference (ww.eval.org), for example, featured over 30 sessions involving the topic of developmental evaluation. Here is a recent curated set of resources on DE from the folks at Better Evaluation http://betterevaluation.org/blog/developmental_evaluation. It is certainly one of our service offerings at FSG, but we are by no means the only ones in the field talking and writing about DE as a nonprofit organization ourselves, we are deeply invested in furthering the evaluation field (and the social sector) by educating and building capacity on various evaluation topics, irrespective of whether it leads to direct consulting work for us or not. 2) Misinformed or misleading conclusions about the Microfinance study: While you correctly imply that we neglected to mentioned tea and food consumption, the report combines these with "temptation goods" such as alcohol, tobacco, and betel leaves (p.19 and p.28). In fact, the tea and food consumption refers to those consumed outside the home, and hence considered "temptation goods". We would assume that cutting back on these would indeed be desirable. Also, the difference between 12 Rupees and 5 Rupees, while negligible in dollar terms, is referred to by the report's authors as "striking" (p.19). Our other claim about a "third more businesses" is also derived directly from the report (p.17) where the authors suggest, "In comparison areas, 5.3% of households opened a business in the year prior to the survey, compared to 7% in treated areas, so this represents 32% more businesses in treatment areas than in comparison". Overall, we want to re-emphasize that the bigger picture issue we are trying to raise here is what the right tools are for evaluating social innovations. It is not about putting the welfare or the project above the welfare of the beneficiaries, but it is about ensuring that a new innovation has time to learn, adapt and fully form, before being evaluated through a summative RCT-type approach. We are urging nuance in a place where absolutes have tended to rule, and we believe that Developmental Evaluation is such a nuanced and comprehensive approach. However, just as RCTs have their time and place, so does DE, and we would in no way suggest that it is the right approach to every situation.
The choice of microfinance and education in these countries is ironic. Perhaps laptops improve certain abilities, and perhaps microfinance has additional benefits the RCTs fail to capture. But perhaps microfinance adversely impacts child education? Various academic papers have suggested that microfinance may encourage children to quit school, or parents to withdraw their children from school to work in the micro-business. Is it therefore a coincidence that Peru is saturated with microfinance, has a huge informal economy where child labour is rampant and MFIs offer their loans, and has the highest child labour rate in Latin America (jointly with Bolivia)? Could an unintended consequence of microfinance be to jeopardize MDG#2 (universal education)? We are all familiar with the kids stacking shelves in crowded markets during school hours - is the solution therefore to give them all a laptop? Perhaps tighter regulation of microfinance is a wiser avenue to pursue. MFIs should lend only to enterprises where they are reasonably sure than child labour is not used in the business, which is not hard to assess. The Smart Campaign, a self-regulatory mechanism designed to clean up the ailing microfinance sector via endorsement of a set of ethical principles (but not actual enforcement) carefully ignores the rights of children in its oft-touted Client Protection Principles. Thus the situation may be even more complex than this article alludes to. But one ought to be concerned by Jairam Ramesh's recent summary of Indian microfinance. As minister for rural development his opinion bears taking seriously. He described it simply as "crap", and suggested: "Nothing can stop an idea whose time has gone. And micro-finance is in a danger zone. It is a discredited model. It has raised more questions that it has answered". Perhaps not as rigorous as an RCT, but are we to ignore his comments entirely? http://www.business-standard.com/india/news/micro-finance-is-in-danger-zone-ramesh/197633/on
Dear Hugh, Thanks for raising this important issue. While we agree that microfinance can and does have unintended consequences, the bigger issue we are raising is around the right approach to evaluate any social innovation. While RCTs have benefits in certain circumstances, using an RCT alone as a tool to evaluate social innovations is problematic, as it may cause us to "throw the baby out with the bathwater" when it comes to potentially promising innovations. A different approach such as Developmental Evaluation would take into consideration the context and implementation, as well as unintended outcomes, and emphasize what we can learn from the data. Rather than endorsing or discrediting the whole microfinance "model", the developmental evaluation would identify ways to shape and strengthen it before it is ready to be evaluated in more traditional ways.
 
 
FSG
FSG is a nonprofit consulting firm specializing in strategy, evaluation, and research. The firm was founded in 2000 as Foundation Strategy Group by Harvard Business School Professor Michael E. Porter and Harvard Kennedy School Senior Fellow, Mark Kramer. Today, FSG works across sectors in every region of the world—partnering with corporations, foundations, nonprofits, and governments to develop more effective solutions to the world’s most challenging issues. FSG’s ideas are frequently published in journals such as Forbes, Harvard Business Review and Stanford Social Innovation Review.
 
 
 
Most Popular
FSG's Activity Feed
February 26, 2013 08:10 am by My Homepage
... [Trackback]... [...] Read More here: forbesindia.com/blog/the-good-company/fix-that-fits-what-is-the-right-evaluation-for-social-innovation/ [...]...
December 05, 2012 20:33 pm by Irvin Gomez
OLPC Peru is certainly not the best example for the points the author tries to make. OLPC Peru is a very sad case of poor people whose very limited resources have been wasted (due to government corruption) on a pseudo-scientist's demented education dreams ("we give a kid a cheap laptop, he won't...
FSG
FSG
December 04, 2012 16:50 pm by FSG
Thank you for your comments. We would be happy to respond to the points that you have raised: 1) Truth in advertising: Developmental Evaluation is a legitimate field within the broader evaluation field, with several consultants and practitioners actively involved in DE efforts. DE is especially g...
December 04, 2012 14:48 pm by Phil
Truth In Advertising - Yes, Advertising After reading this article - which appears rather a PR effort to sell a consulting product than a real contribution to important questions - the mind boggles about the added value of the "developmental evaluation" which is advertised here. While much is wro...
FSG
FSG
December 01, 2012 10:35 am by FSG
Dear Hugh, Thanks for raising this important issue. While we agree that microfinance can and does have unintended consequences, the bigger issue we are raising is around the right approach to evaluate any social innovation. While RCTs have benefits in certain circumstances, using an RCT alone as ...