A call to evaluate social innovation. . .differently

This post originally appeared on FSG's Social Impact blog.

Question: What do the following foundation led initiatives have in common?

1. The J.W. McConnell Family Foundation’s YouthScape program tested strategies for involving excluded youth in the community development process, in part by encouraging traditionally adult-focused organizations to examine and re-design their values, structures, and processes.

2. The John S. and James L. Knight Foundation’s Community Information Challenge (KCIC) funded 76 community and place-based projects, to create and support new news models, online information hubs, citizen journalism activities, youth media outlets, advocacy campaigns, and civic engagement activities, in an effort to keep communities informed and engaged.

3. The Bill & Melinda Gates Foundation’s Community Partnerships portfolio focuses on improving post-secondary completion rates for low income young adults through the activation and coordination of a number of diverse stakeholders. Community partnerships use data to inform their strategy, they seek to build commitment among stakeholders, and they tackle policy and practice change.

Answer: They are all social innovations!

As solutions to social problems have become more complex and challenging, the philanthropic and non-profit sector has been experimenting with a variety of approaches such as systems building, policy advocacy, cross-sector collaboration, movement and network building, and collective impact, as a means to affect change more rapidly and effectively.

While many organizations are commissioning formative and sometimes, summative evaluations of these initiatives, we are learning that traditional evaluation approaches are a poor fit for creative and innovative programs where there is no accepted model, where the path to success is unclear, and the outcomes are not completely understood or known. What is needed instead, is an evaluation process that provides ongoing information, works in close partnership with the initiative’s implementers, uses multiple communication strategies, and adopts a systems orientation. The J.W. McConnell foundation has embraced this approach for several years:

Many of the McConnell foundation-funded initiatives work in uncertain territory, developing and testing their strategies as they proceed; there are no blueprints for empowering youth, attacking poverty, or promoting innovative approaches to solve entrenched social problems. What is most useful for such efforts is not an ex post facto assessment of success or failure, but constant feedback from a critical, supportive observer.

This month, FSG and the Center for Evaluation Innovation, will be releasing a new report titled, Evaluating Social Innovation. The paper is the result of a collaborative research study that sought to understand and identify the critical role evaluation plays in initiatives that are innovative, situated in dynamic and complex environments, and are being designed and implemented simultaneously. In the paper, we describe and make a case for the use of developmental evaluation for evaluating collaborative, emergent, system change processes.

Originally conceptualized and described by evaluator Michael Quinn Patton, developmental evaluation asks evaluative questions, applies evaluation logic, and gathers and reports evaluative data to support project, program, product, and/or organizational development with timely feedback.

To this end, developmental evaluation typically focuses on the following kinds of questions: > What is developing or emerging as the innovation takes shape?
> What variations in effects are we seeing?
> What do the initial results reveal about expected progress?
> What seems to be working and not working?
> How is the larger system or environment responding to the innovation?
> How should the innovation be adapted in response to changing circumstances?

The answers to these questions help program designers and implementers reflect on the data, challenge their assumptions, and make decisions regarding staying the course or shifting tactics and strategies in timely ways (not at the end of a year or two or three).

In Evaluating Social Innovation, we explore ways that common evaluation approaches and practices constrain innovation and consider the following questions: What kinds of grantmaking strategies should funders consider using developmental evaluation? What organizational conditions are necessary for developmental evaluation to work? How can grantmakers grapple with the challenging questions that developmental evaluation raises about innovation, accountability, rigor, and adaptation?

Drawing on the reflections and insights of foundation staff and evaluators who have experimented with developmental evaluation, we call on philanthropy to re-envision the role, purpose, and processes of evaluation so that social innovations may have an even greater chance of achieving their ambitious goals.

By Hallie Preskill, PhD, FSG Managing Director
(Hallie oversees FSG’s Strategic Learning & Evaluation approach area. Prior to joining FSG, Hallie spent more than 20 years in academia, teaching graduate level courses in program evaluation and organisational learning, among others. She has authored several books and articles on evaluation related topics, and served as the President of the American Evaluation Association in 2007.)

The thoughts and opinions shared here are of the author.