Close
Design sprint workshop at TICTeC at the OECD in Paris
Design sprint workshop at TICTeC at the OECD in Paris

Designing Social Impact Metrics for Civic Technology

by Alisa Zomer, Erhardt Graeff, and Kelly Zhang

Bringing together partners from the US (POPVOX) and South Africa (Grassroot), we tested a new format to co-develop metrics and civic technology in an iterative design process.

This year’s largest civic technology gathering of practitioners and researchers (#TICTeC) took a hard look at the state of civic technology (i.e. Are we now in the fourth wave? Is it a sea change movement or has civic tech lost its relevance?). Focusing on questions of measuring impact, our team, including Erhardt Graeff from Olin College and Alisa Zomer and Kelly Zhang from MIT GOV/LAB, put together a design sprint workshop with partners POPVOX (U.S) and Grassroot (South Africa). The aim of our session was to kick-start an iterative design process to integrate social impact metrics into platform design from the very beginning, not as afterthought.

What does this mean in practice?

Erhardt Graeff at TICTeC 2019
Erhardt Graeff at TICTeC 2019.

As Erhardt discussed in recent work, impact for civic technology is often measured using basic descriptive data from the platform itself, including number of users, repeat users, time spent on the platform, etc.; however, these engagement metrics mostly fail to speak to the social impact goals at the heart of civic technology. In some cases, surveys are conducted where users rate their experiences on a platform, which can be helpful for troubleshooting user experience design problems. But answering the real question of social impact (i.e. has the platform changed outcomes, either beliefs or behavior) demands a different kind of metric.

In our workshop, we introduced an approach that starts by articulating what change you want to see—do you want to empower ordinary citizens to take a particular action (e.g., petition, protest, vote)? Or, do you want to see local government have a specific response (e.g., budget allocation, new legislation, improved service provision)?

Depending on the intended change, there is often relevant research to help in the design process. A wealth of research exists on what incentivizes or prevents citizens from engaging with their government, how to build strong grassroots movements, and also what barriers or opportunities present on the government side. Building on this knowledge, our goal is to bridge theory with design in a way that aligns with achieving and measuring social impact.  

For POPVOX, a platform to improve dialogue between citizens and government in the US, they ask: How might we measure participants’ understanding of government processes, comfort engaging, and sense that their voice matters? One approach taken by Erhardt previously is to measure perceived political efficacy (or the belief that you can influence or affect political change) in his work with SeeClickFix. He adapted a number of oft-used national survey questions for internal and external political efficacy to examine the context of city residents requesting their local governments fix the things they care about, finding correlations between government responsiveness on the platform and active users’ perceptions that their local governments were listening to them.

Screenshots from Marci Harris’s presentation on POPVOX’s platform
Screenshots from Marci Harris’s presentation on POPVOX’s platform.

Grassroot, which provides low-tech, low-cost tools for grassroots organizers working primarily with low income groups in South Africa, posed the following: How might we measure the ways in which social movement organizations grow and build capacity? Their main design constraints include a consistently unresponsive government and social movement organizations that have strong initial momentum but then steeply decline.

Screenshots from Luke Jordan’s presentation on Grassroot’s platform
Screenshots from Luke Jordan’s presentation on Grassroot’s platform.

Dividing into small groups, we tackled these design prompts to come up with relevant and feasible metrics, explain why they matter, and what data would be needed to measure them.

Sprinting for impact

Design sprinting in 70 minutes is not for the faint of heart, but we forewarned participants and the end results were a good start to a longer process. For Grassroot, some of the suggestions included looking at leadership development within organizations over time, assessing media visibility as a proxy for reach outside the movement, and measuring interactions between organizers across multiple policy issues. For POPVOX, suggestions included design features that notified citizens when officials read comments in order to build in some measure or response and trust-building.

In the coming months, both teams plan to move these pilots forward, building on the outcomes of the TICTeC workshop.

Workshop descriptions and slides available online. For more information, contact the MIT GOV/LAB at mitgovlab@mit.edu or Erhardt Graeff at erhardt@erhardtgraeff.com.

All event photos from Brian David Films

This report is cross-posted at the MIT GOV/LAB blog.

Holding Civic Tech Accountable for Civic Learning

Presentation Recording

Slides

http://web.media.mit.edu/~erhardt/slides/DML2016-Ignite-Graeff-CivicLearningTalk.pdf

Abstract

We should optimize the design of civic technologies for developing effective citizens; I argue we must put civic and political learning at the core of how we evaluate civic technology’s impact. This will require new definitions and measures that capture the complexity and needs of contemporary, digitally-mediated democracy.

In this talk, I will propose a research agenda for civic learning including definitions, measures, and design goals for our community to explore.

Recent work by Bennett (2007) and Cohen and Kahne (2012) has helped push scholar and practitioner communities to understand how citizens, especially younger generations, are changing in their civic goals and practices—often using social media to consume and share political information, express their voices, and organize civic and political communities. Unfortunately, we lack a definition and operationalization of how “users” grow into the citizens contemporary democracies need. Research should be attempting to tie designs holistically to gains in targeted skills, experience, and self-efficacy.

We must develop and validate measures for civic learning by combining rich qualitative understanding with trace data to evaluate users’ civic trajectories as they explore tools and platforms; and these must scale as large as Facebook and across the diverse contexts in which users are embedded world-wide.

From User to Citizen: Evaluating Online Engagement as Civic Learning

Presentation Recording

Slides

http://web.media.mit.edu/~erhardt/slides/TICTeC2016-CivicLearningTalk.pdf

Abstract

We should optimize the design of civic technologies for developing effective citizens, thus I argue we must put civic and political learning at the core of how we evaluate civic technology’s impact. This will require new definitions and measures that capture the complexity and needs of contemporary, digitally-mediated democracy.

In this talk, I will propose a research agenda for civic learning including definitions, measures, and design goals for our community to explore, using existing literature and analysis of a pilot deployment of the civic app Action Path.

Recent work by Bennett (2007) and Cohen and Kahne (2012) has helped push scholar and practitioner communities to understand how citizens, especially younger generations, are changing in their civic goals and practices—often using social media to consume and share political information, express their voices, and organize civic and political communities. Unfortunately, we lack a definition and operationalization of how “users” grow into the citizens contemporary democracies need. Research should be attempting to tie designs holistically to gains in targeted skills, experience, and self-efficacy.

We must develop and validate measures for civic learning by combining rich qualitative understanding with trace data to evaluate users’ civic trajectories as they explore tools and platforms; and these must scale as large as Facebook and across the diverse contexts in which users are embedded world-wide.

In partnership with SeeClickFix in New Haven, I recently piloted Action Path, a location-based mobile app that invites users to engage in local planning and governance via push notifications and short surveys. I interviewed users and traced their app usage to evaluate the efficacy of location-triggered notifications for increasing knowledge of local issues and engagement in governance. I will highlight the implications of this deployment for civic learning and how it represents a small start to the research agenda I am proposing.

Civic Media Impact

Presentation Recording

Slides

https://erhardtgraeff.com/portfolio/file_download/17/BCMC-CivicImpact-Intro.pdf

Workshop Description

Goals: To share approaches for understanding impact in civic media – from randomized controlled trials to qualitative evaluation to artistic research methods. The workshop will explore the variety of ways in which people are responding to the increased need to demonstrate the impact of media interventions, including how we evaluate the impact of research, practice, and classroom-community partnerships.