Skip to main content

Legislating peer review? A letter to a policy maker

Rep. Lamar Smith (R-TX)
Chairman, House Science, Space, and Technology Committee

Dear Chairman Smith:

As you continue to shape the High Quality Research Act (or whatever its final title), a bill concerning peer review at the National Science Foundation, I urge you to ask the following questions:

1. Is the intention to produce greater transparency about the merit review process? 

As a former congressional staffer, I appreciate the need for oversight of the executive agencies within your purview. But as a former NSF division director, I also respect the need for discretion in pursuit of its mission and the conduct of agency business. 

As you know, at NSF peer review is governed by two criteria—intellectual merit and broader impacts. How these are fulfilled is judged by technical experts acting as proposal reviewers whose role is confidential and advisory. But the way these reviewers are deployed varies greatly across and within directorates, divisions, and programs.   

Despite these variations, NSF has over the years examined its processes and performed its own internal audits of the quality control that peer review represents. How much it wishes to share with its authorizing committee is also a matter of judgment. In my view, more is better. How much transparency is enough?

2. Are you questioning the stewardship of NSF staff?

I would hope not because the authority delegated to NSF has worked admirably for 60+ years. This is not to say that over 1,000 program officers, division directors, and assistant directors —many who rotate in and out of the agency—have been unassailable. But the historical record is one of due diligence and consistent productivity.

In fiscal 2011, NSF reviewed—by mail, panels, and committees—51,000 proposals. It funded 11,000 (or 22%) of them. Failure through competition, one could say, is the norm (and it takes significant labor). The time and energy consumed in the reviewing process is huge (over a quarter million independent reviews were submitted). Budget constraints only raise these stakes. And after all this, the National Science Board reviews the biggest budget items that have been recommended for funding. 

In short, with multiple layers of "eyes" on proposed work, the funding process is anything but capricious or ill-considered. Yes, this is done largely out of the public eye. Again, transparency is measured, but accountability is served.

3. Is the concern really with the outcomes of the process, i.e., the decisions about what to fund?

If so, then your motivation for putting forth the bill would seem to be divorced from questions 1 and 2 above. Rather, you may disagree with some awards because you find them inappropriate for NSF support. They may, in your view, lack a direct link to health, security, economy, or any other national goal that you deem a priority. The Coburn Amendment has already imposed new criteria on the review of political science proposals and NSF has sadly capitulated to this arbitrary targeting of one discipline.

Is this your committee's objective? The existing two NSB-established criteria already encompass the goals above. It is the responsibility of reviewers to judge a proposal's adequacy in achieving them. It is for peers (practicing scientists) to decide, not authorizers (elected officials who are nonetheless non-peers). 

Yet as NSF's congressional overseers, you should expect responsiveness to questions about how the agency does business. You should not expect that the minutiae of its decision-making processes become an open book. For the integrity of those processes depends in large part on non-disclosure and respect for, indeed trust in, them.

So it is in federal grants-making as in other aspects of public life. You shouldn't divulge everything. And you can't legislate peer review. 

Thank you for your continued vigilance.
Daryl Chubin
A Concerned Citizen 

Related Links:

 

Date
Blog Name