Earlier this year, NSF solicited reaction to proposed change in its "broader impacts" criterion (also known as Criterion 2). Yet concerns have been raised about the National Science Board-mandated criterion for over a decade: What does broader impacts mean? How inclusive is it -- education at all levels, public outreach, participation of underrepresented groups in science and engineering, mentoring, student research experiences, etc.? Along with "technical merit," reviewers must judge promised applications of research findings or social benefits that flow from NSF project funding.
In the view of many Criterion 2 had no real impact on funding decisions, indeed that in a silent conspiracy of program directors and reviewers the broader impacts were either ignored or accorded no more than marginal weight in the review process. Some research communities devised ways to circumvent the criterion by using enough reassuring words in their proposals to satisfy the letter, but hardly the spirit, of the criterion.
Some of us held our breath when the National Science Board (NSB-11-42) issued a June 14, 2011 "Dear Colleague" letter. It stated: "Collectively, NSF projects should help to advance a broad set of important national goals, including:
- Increased economic competitiveness of the United States.
- Development of a globally competitive STEM workforce.
- Increased participation of women, persons with disabilities, and underrepresented minorities in STEM.
- Increased partnerships between academia and industry.
- Improved pre-K—12 STEM education and teacher development.
- Improved undergraduate STEM education.
- Increased public scientific literacy and public engagement with science and technology.
- Increased national security.
- Enhanced infrastructure for research and education, including facilities, instrumentation, networks and partnerships."
Additionally, "Broader impacts may be achieved through the research itself, through activities that are directly related to specific research projects, or through activities that are supported by the project but ancillary to the research."
In response, performers (including AAAS, in a July 13, 2011 letter from Malcom et al. to the National Science Board Chairman and the NSF Director) registered alarm over what will be lost or how funding decisions will be narrowed, under the revised Criterion 2. And in July, the House Science, Space, and Technology Subcommittee on Research and Science Education held a hearing to examine the system within which broader impacts are embedded -- merit review.
One particular protest of broader impacts comes from Nature columnist and long-time policy analyst Daniel Sarewitz. Writing in a July 14, 2011 editorial, "The Dubious Benefits of Broader Impact," Sarewitz assails "the notorious Criterion 2."
Sarewitz raises three concerns. First, he questions the ability of panels to recommend which claims of social benefit are "more or less important." In doing so, he doubts the qualifications of reviewers to make such judgments. By doing so, he lets scientists as citizens -- rather than as experts -- off the hook. Playing the former role is implied by the broader impacts criterion.
Second, Sarewitz raises the policy and administrative issue of whether "individual projects" are the "wrong lever to bring NSF research into line with national goals." We could debate whether broader impacts would be better judged at the aggregated level of a program. But that would undermine the accountability of individual investigators for NSF investments. A scheme would be needed to apply the broader impacts criterion to some projects and not to others—a sampling and measurement problem, to be sure.
Finally, Sarewitz's argument would require the sponsor to (re)educate research performers and to provide a reasonable standard for fulfilling the criterion. In an agency like NSF, which fancies itself as field-driven, this is difficult at best.
Policy is full of tradeoffs. Of any astute policy analyst, we should ask: Can merit review be updated or does it need an overhaul (a question now facing NIH)? How does NSF reconcile national accountability with highly decentralized processes of review and research performance? And ultimately, is no lever that focuses on the benefits of research impacts as a funding criterion better than an imperfect lever?