What does evaluation mean in times of COVID-19?

By: Juan Carlos Piña

In response to the rising worldwide spread of COVID-19, on March 11, 2021 the World Health Organization (WHO) declared a global pandemic. WHO’s announcement of the imminent threat that COVID-19 presented for the nation’s public health led to the abrupt closures of offices, workplaces, and schools, among other locations. The range of site closures where people conduct business, work, learn, and play changed life and human interaction as we know it. There has been a slew of hardships experienced by families, including housing and food insecurity, job losses, increased student learning gaps, and reduction of available childcare.

Because state and local supports to the crisis have varied, community-based organizations and government agencies have necessarily shifted their strategies to donate their time, talents, and financial resources to provide basic needs for communities, particularly those most impacted by COVID-19. Over the course of the past 12 months, SPR staff have worked with partners to understand the impacts that COVID-19 has on our communities, lift up learnings, and facilitate strategic planning for future efforts to continue providing ongoing support. Through survey responses, interviews and focus groups, our staff have been deeply connected to these issues through the personal stories of people who have been affected by the pandemic.

While many of us are still thinking through the long-term implications, we asked our staff to pause, reflect, and share their thoughts on how the COVID-19 pandemic has influenced how they see their role as social science researchers in the field of evaluation during a pandemic. The following are reflections from some of our colleagues in SPR’s Equity, Education & Community Change Division.

 

Centering community perspective for an equitable COVID-19 recovery.

While COVID-19 has shifted aspects related to the “how” of SPR’s work (e.g., working from home, conducting data collection virtually, and finding creative ways to collaborate with our partners in virtual spaces) in many contexts, the “it” of our work has not shifted, but only intensified with the pandemic. In reflecting on my work as an evaluator during 2020 and into 2021, I have been most influenced by seeing how the pandemic has widened existing inequities faced by communities of color, and I have been trying to understand how COVID-19 relief can directly address the structural racism that caused the inequities in the first place – particularly in the contexts of early care and learning and housing. In both contexts, COVID-19 has brought a sense of urgency and political will to infuse resources supporting recovery from the pandemic. However, there is a need to address longstanding inequities in addition to immediate COVID-19 relief. As SPR has worked in these contexts throughout COVID-19, I have seen my colleagues double down on partnering with community to support them in bringing their perspectives to these COVID-19 relief efforts. We have been innovative and relentless with ensuring that community members most impacted by COVID-19 can participate in focus groups, interviews, workshops, and surveys—recognizing that, without their perspectives, relief efforts may not address longstanding inequities. My hope is that through centering these perspectives, COVID-19 relief can support impacted communities with not only recovering from the pandemic, but also with sustainably thriving in a system that truly works for them.

– Laura Pryor, Senior Associate

 

Serving as a learning partner in uncertain times.

The COVID-19 pandemic and events of 2020-2021 have challenged our project teams to be more flexible and responsive than ever, while confirming my deepest convictions about what it means to be an evaluation and learning partner. My colleagues and I have now spent a year walking alongside our partners as they take on new roles, experiment with providing services in different ways, and respond to the vastly expanded needs of the children, families, and community members that they serve. Along the way, we have shifted our evaluation questions, our approach to data collection, and our reporting strategies to meet the needs of the current moment and to be sensitive to changing conditions on the ground. This has reinforced qualities that SPR has always sought to bring to our evaluation partnerships, such as humility, self-reflection, humor, adaptability, tolerance with uncertainty, and responsiveness. I have also been inspired by the unprecedented convergence around issues of equity in our projects, as our partners grapple with the disproportionate impact of the pandemic on BIPOC communities and on biases within their own organizations. Above all else, over the last year, I have been awed by the resilience of our communities, clients, and collaborators and am honored that they continue to entrust us to tell their stories.

– Heather Lewis-Charp, Division Director

 

Reviving our purpose of contributing to a society where everyone can thrive.

When I think about the year 2020 and the impact COVID-19 has had on the world, I cannot separate it from the rising public awareness of racial inequity. It makes me shake my head to think that some of us ever thought of the virus as “the great equalizer.” Of course, now we know the pandemic has not impacted everyone equally, but instead has exacerbated existing social and economic inequality. At SPR we work with advocates and community partners who have been warning that there is a lack of a safety net in this country. For many individuals it just takes a job loss or a large medical bill to send them into a downward spiral. In this work, these were things we knew, but COVID-19 made these inequities undeniably obvious, even for people not in the field or not working directly with low-income and BIPOC populations.

Early in the pandemic, I must admit I felt a bit of an identity crisis as an evaluator. As our programs and partners swiftly reacted to the fallout of the pandemic, it wasn’t clear where the evaluation plans we had worked so hard on fit into the picture. However, the chaos of the initial months of the pandemic settled, and conversations slowly shifted to the opportunity to rebuild our systems better. As 2021 continues, I feel hope and a renewed sense of purpose for our work providing data and feedback on policies and programs that work to remedy inequities.

– Savannah Rae, Associate

 

Understanding evaluation as a vehicle to advance racial justice.

2020 was a challenging year on many fronts. Not only have we seen the disproportionate impact of COVID-19 on Black and Brown communities, the murders of Breonna Taylor, Ahmaud Arbery, and George Floyd and the subsequent racial justice uprisings continue to show the pervasiveness of racism and white supremacy in our systems. At the same time, through our evaluation projects, I have been honored to be in community with numerous organizations across the country pushing for racial justice through advocacy and systems change. In this last year, I have spoken to numerous leaders of color through interviews, focus groups, learning convenings and sessions, with the aim of deepening areas of inquiry on how COVID-19 and the racial justice uprisings have impacted our communities, captured progress that has been accomplished, and lifted insights around the future direction of our society. As an evaluator, I have always seen my role inextricably linked to my racial and gender identity. Through this process, as a first-generation Latina from rural California, this last year allowed me to not only speak to people that were impacting communities like mine, but also proved that evaluation is a powerful tool for advancing racial justice.

– Verenice Chavoya-Perez, Senior Associate

 

The overarching lesson shared by our evaluators at SPR has been to examine the current environment that we live in and adopt new approaches all together. In doing so, we have identified new opportunities by shifting existing evaluation practices to meet the needs of our communities and partners. And within that, many have rekindled motivations for the field of evaluation and identified the “so what” of why they are doing the work to begin with.