February 11, 2020

Measurement can be messy, especially in the impact space! As we’re witnessing an explosion in efforts to deliver social change, we’re also experiencing how difficult it is to track performance toward these worthy and lofty social goals.

Beeck Center leader Nate Wong moderated a discussion with leading impact measurement expert Alnoor Ebrahim and Miriam’s Kitchen CEO Scott Schenkelberg on Tuesday, February 4, 2020 at Georgetown University based on the release of Ebrahim’s new book, “Measuring Social Change: Performance and Accountability in a Complex World.” The rich conversation covered both theory and practice, recognizing how difficult it is to translate strategy to reality. This conversation was part of the Georgetown University’s Center for Public and Nonprofit Leadership class taught by Professor Kathy Kretman.

cover image of Measuring Social Change
Get the Book [Stanford University Press]
Three key takeaways are:

  • To start, we need to actually confront our assumptions of what are “good” metrics. While there’s a strong emphasis on long-term outcomes, “there are certain kinds of outputs where short term outputs make a great deal of sense (video),” Alnoor exhorted the audience in response to Nate’s question. Think about ambulances. Service delivery times are incredibly important – they are a matter of life and death… literally. So it’s more of a question around what do these metrics help drive? It’s more about strategy, performance, and accountability.
  • To measure social change, it first starts with the organization’s strategy and their theory of change. Organizations like Miriam’s Kitchen can shift strategies, in this case from a niche to an ecosystem strategy, but it takes a very different type of approach and skills. It requires radical collaboration. Miriam put a bold mission forth to end chronic homelessness in Washington, D.C., which dramatically reconceptualized how they operated. Instead of focusing on meals and programs, they needed to orchestrate the over 100 other players around joint goals and coordinated action. “I was skeptical at first (video),” Miriam’s Kitchen CEO Scott Schenkelberg admitted. “We were really good at delivering outputs, think of our meals for the homeless and holding onto our select sphere of influence. Going out there and saying that we can end chronic homlessness, that’s a big scary statement… When I say that we are going to end chronic homelessness, we better end it, or else it’s a really big failure. People said it’s going to take time Scott; here are the things that we need to do, Let’s explore this process by which we can do it so.”
  • The measurement strategy is only as good as the system that it sits within. As Ebrahim points out (video), “The funder ecosystem is highly fragmented, and there is a challenge for funders to be doing more in terms of collaborative fundraising, so there’s sufficient pools of resources to sufficient pools of executing organizations to address social problems at some sort of scale.” Making this move will entail shifting funders to articulate coordinated theories of change with a shared sense of objective measures. It will also take incentives for coordination and collaboration among implementers.

Ebrahim is a professor at The Fletcher School of Law and Diplomacy at Tufts University, whose current research addresses two core dilemmas of accountability facing social enterprises, nonprofit organizations, and public agencies: How should they measure and improve their performance? How should they address competing demands for accountability from diverse stakeholders?

The event was sponsored in part by McCourt School of Public Policy, Center for Public and Nonprofit Leadership, Beeck Center, Georgetown College, and the Social Responsibility Network (SRN) of Georgetown College.


Read more about the event from the Center for Public and Nonprofit Leadership – Measuring What Matters: Strategies for Measuring Change in the Social Sector


This is one of the many events that the Beeck Center hosts throughout the year, everything from workshops to open houses to author events like this one. They are an excellent opportunity to bring together members of the student and broader social impact communities for spirited conversations on important issues. Previous guests include Ann Mei Chang, author of “Lean Impact,” and Anand Giridharadas, author of “Winners Take All: The Elite Charade of Changing the World.”

Join us on March 31 for our next author event, as Henry Ramos discusses his new book, “Democracy and the Next American Economy.”

December 20, 2019 | By Margarita Arguello

As anyone choosing a college knows, finding the right balance of academic strength and affordability is incredibly difficult. In September 2015, in the midst of rising college costs and a changing U.S economy, the Obama administration launched the College Scorecard — a tool to help American families make decisions about college attendance by providing them “clear, reliable, open data on college affordability and value.” 

By pooling together data from various government agencies and conducting hundreds of interviews with potential users, government teams from the Department of Education, 18F, and the United States Digital Service (USDS) created something so useful, that within a year, 1.5 million unique users had accessed it and Google integrated its data so that it appears front and center when users conduct college-related searches. 

screen capture of google search for Georgetown University
In this screen capture from a Google search, the data fields including “Average cost after aid,” “Graduation rate,” and “Acceptance rate” draw from data collected by the U.S. Department of Education that is made available as open data through the College Scorecard project.

The College Scorecard is one example of how, throughout the past decade, the U.S. government has worked to leverage the tools of technology, data, and design to create digital products and services that directly impact the lives of people across the country. 

As evidenced by the College Scorecard, data and digital are already proving to be powerful drivers of social impact. However, there is still much to figure out in terms of how to best use these tools to create solutions that actually work for people, while also being mindful of user privacy and safety concerns. This is what the Beeck Center’s Digital Service Collaborative and the Georgetown Technology Policy Initiative sought to explore this fall through its Data, Digital + Social Impact Seminar Series. Over five seminars, experts in the field joined Georgetown students to deconstruct the nuances behind digital transformations in government  and delved deep into the topics of open data, privacy, ethics, and design for social good. Providing students an opportunity to learn practical solutions for delivering positive social impact is a key element of the Center’s mission, and one way it generates impact at scale.

The kick-off seminar provided students with a basic understanding of the building-blocks of data and examples of how social impact leaders have worked to improve the lives of people around the world through the use of open data initiatives that is, by releasing data to the public in digestible formats that allows anyone to access the data, use it and republish it. For example, Beeck Center Fellow Denice Ross, who was previously a White House Presidential Innovation Fellow and before that led open data for the City of New Orleans, talked about how emergency planners in the city of New Orleans now use open data from various federal and state government agencies to better coordinate their hurricane response efforts to prevent disastrous scenarios like those seen in the aftermath of Hurricane Katrina. 

Similarly Erica Hagen, co-founder and director at GroundTruth Initiative,  spoke about her experience leading Map Kibera — a project that aimed to create a map of Kibera, a previously unmapped slum in the Kenyan city of Nairobi. By using GIS mapping technology and enlisting the help of the people of Kibera, Hagen and her team were able to produce a map giving residents of this neighborhood access to accurate and useful information, such as the location of clinics and pharmacies within the community.

These conversations raised the important question of how to use data to design policy solutions that actually work for the people they are intended to serve. During later seminars, the expert speakers weighed in on this issue and discussed basic frameworks for effective policy making in an era of rapidly changing digital technologies. 

 

Students sit around a conference table
Beeck Center Fellow Natalie Evans Harris discusses privacy implications of data use on Oct. 8, 2019.

The most prominent ideas that the experts proposed for solving this problem involved adopting some of the popular practices of the tech sector into the policy-making process. These include conducting thorough rounds of user research in the early stages of the policy-making process, designing experimental solutions that can eventually be brought to scale, creating a “Minimum Viable Product” and then iterating to improve it, and using tech-enabled methods to incorporate the views, habits and feedback of the people that their products and services are intended to help. 

Throughout every session of the series, the experts walked students through case studies that exemplify how these concepts have been applied in real life. For example, USDS founding member Erie Meyer, who is now a tech advisor at the Federal Trade Commission, talked about how, in leading the College Scorecard project, her team took a user-centered approach that involved conducting extensive user research and collecting feedback along every step of the process. 

Likewise, Rebellion Defense co-founder Chris Lynch spoke about his experience working to improve the lives of U.S. troops in Afghanistan through his work leading the Defense Digital Service at the U.S. Department of Defense. In this role, Lynch and his team revamped and reinvented the tools and practices that the DoD uses on the ground.

Although the conversations in these five seminars shed light on the complex processes that have made digital solutions like the College Scorecard a promising reality in the United States, much work remains to be done as policy makers and scholars alike navigate the process of digitizing social services. For this reason, the Beeck Center continues to explore the topics of data governance, digital service delivery, and tech policy through its Data + Digital portfolio. 

 

Margarita Arguello is a student analyst at the Beeck Center for Social Impact + Innovation. She is a master’s student in the McCourt School of Public Policy and will graduate in May 2020.