AAAS team seeks input on evaluating impacts of public engagement on scientists

Scientists engage with members of the public in a wide range of settings and formats, which can pose challenges for evaluating success.| Credit: Atlantic Photography Boston AND ARCHITECT OF THE CAPITOL.

Two recent papers take important steps toward evaluating the impact of participating in public engagement on scientists. Most evaluation of public engagement with science has focused on impacts on the public participants. Assessing the effects on scientists could help document the mutual exchange of ideas that is central to the public engagement approach. Such evidence could provide concrete data and examples for scientists of what they can gain from discussing science with members of the public, not just what they can provide, and help them justify their involvement.

The AAAS Center for Public Engagement with Science has been working with researchers and evaluators Karen Peterman, Jane Robertson Evia, and John Besley to measure these impacts on scientists. The scales they’ve developed could be used as part of evaluation instruments (such as surveys) that scientists complete after individual public engagement activities or periodically during more ongoing public engagement efforts. AAAS plans to use these scales in its programs -- for example, surveying the scientists that participate in Family Science Days and providing the scales to fellows of the Leshner Leadership Institute for Public Engagement with Science, as a common but adaptable way for them to evaluate the programs and activities they organize. AAAS also plans to make these scales widely available for others to use in assessing the impacts of public engagement.

The evaluation stems in part from the AAAS logic model and theory of change for public engagement with science, which describes short-, medium-, and long-term outcomes for individuals and for society more broadly. Each paper describes how researchers developed and tested scales for one type of potential effect, to determine if the questions asked would accurately assess scientists’ experiences. The researchers tested the questions for each scale through “think-aloud” interviews with scientists, to see if the questions elicited the expected range of responses.

One paper, published in the International Journal of Science Education, Part B: Communication and Public Engagement, described validation of the “self-efficacy” scale that measures a scientist’s belief in their own ability to succeed in a given activity. To use the scale, scientists rate their agreement with 13 statements, including “I am able to create activities that participants find engaging” and “I am able to moderate discussions with participants, even when they include a wide range of perspectives.”

The other scale is for “outcome expectations,” or belief in the effectiveness of the activity itself, and was described in an article in the journal Science Communication. This scale asks scientists to reflect on a specific public engagement activity and rate their agreement with six statements, including “The activity helped participants connect science to their everyday lives” and “The activity provided me with an opportunity to learn from the broader community.”

AAAS and its partners intend to develop an additional scale related to how participating in public engagement affects trust between scientists and members of the public. The team also plans to prepare and disseminate supporting information so other organizations can incorporate these scales into their own evaluation work, and provide feedback to improve them. In the long-term, this effort may expand to include collecting data from across these surveys to draw conclusions about public engagement with science more broadly.

This effort has been discussed on several occasions in the Public Engagement with Science Group on Trellis, the AAAS online communication platform for the scientific community (request to join here). AAAS welcomes anyone interested in learning more about using these scales for public engagement with science to contact Emily Therese Cloyd at the AAAS Center for Public Engagement with Science and Technology (ecloyd@aaas.org).