Social Impact Evaluation

As the field of social impact evolves, practitioners and scholars are building frameworks to evaluate and monitor social impact initiatives. While the natural and social sciences have established standards for assessment, the results of social impact projects and activities are often in service of larger initiatives, making them challenging to quantify and evaluate. The concepts detailed below can help guide the evaluation of these projects.

Standards of Evidence

A helpful framework for conceptualizing evaluation methods are standards of evidence, which categorize metrics of social impact into five levels. 

• Level One 

This level requires a theory of change: a methodology that tracks the ripple effects of social impact work. Though theories of change are necessary for projects, they are often insufficient to prove success. 

 

• Level Two 

This level requires a theory of change as well as data on or evidence of outputs (e.g. number of reports published, people served, skills gained). Though these metrics are quantifiable and show a measurable effect of initiatives, they cannot prove that the project is the primary cause of change. 

• Level Three 

This level requires evidence of a change in the status quo. This means there is a record of the circumstances before and after a project that demonstrates that the conditions have changed. Though this level of evidence is more robust, it still does not prove that the project is the reason for change.

 

• Level Four

This level requires proof that the project was the cause of a change in status quo. This is difficult to achieve because social impact projects do not occur in a vacuum, so it is challenging to create a controlled environment for an experiment.

• Level Five 

In addition to a change in outcome and proof of causality, this level of evidence requires indication that the change is replicable. 

 

Most social impact projects reach level two or three, but few reach level four or above since these require rigorous experiment design and a controllable environment. Even within levels two and three, success of social impact projects is difficult to gauge because of the distinctions between outputs, outcomes, and impact.

 

Outputs, Outcomes, and Impact

Measuring success depends on what metrics are used and whether they measure outputs, outcomes, or impact. 

 

Outputs are the direct, immediate products of an activity. For example, in a project focused on tech-enabled public benefit delivery systems, an output could be a written report. A metric of that output might be the number of reports published. 

 

Outcomes indicate a higher level of success where the project creates a change in the status quo. For example, if a state administrator reads the project’s report and decides to redesign a benefit application, that new initiative is an outcome of the project’s work. This level of success is difficult to evaluate and prove. In this example, the project would have to follow up with every person who reads the report to find out if they implemented its recommendations. 

 

However, even if the project’s work does not immediately result in change, it may cause a positive ripple effect. In this example, a state administrator who reads the report might share it with their network and gain insights that impact future decision-making. While it is not enough to gauge a project’s merits on the hypothetical, many social impact projects focused on local governments require cultural changes and increased awareness and trust of innovative initiatives, so activities that promote such an environment are important.

 

The highest level of success is impact—which is clear, measurable evidence that a significant number of lives have improved. In our example of the tech-enabled benefits report, impact would be the increase in the number of people who received a certain benefit after a new application launched. 

 

As the Beeck Center has learned from our engagements with state, local, teritorial, tribal, (SLTT) and federal governments, many puzzle pieces need to be in place for success: the investment of a high-level decision maker, technical expertise, user engagement, and dedicated staff and resources, among many other factors. Knowledge sharing and documenting examples of this work will help create the enabling environment for projects to succeed. While the end-goal of our work is impact, we can often only measure the outcomes of our work. Crucially, this work towards achieving outcomes is no less valuable. Capacity, culture, knowledge, and investment within governments and their partners are necessary to improve people’s lives. The Beeck Center’s projects contribute to this infrastructure.

 

Conclusion

While social impact evaluation is complex and layered, we are leaning into this challenge guided by the frameworks above. We are working to improve our monitoring and evaluation strategies by identifying what impact looks like and tracing our work through our theory of change. As the field of social impact evaluation advances, we will continue to assess and strengthen our measurement techniques and impact.

Joyce Bai (’24) was a student analyst with the Beeck Center for Social Impact + Innovation during summer 2022.