 |
AAAS Policy Brief: OMB Bulletin on Peer Review
Issue Summary | Resources
Contents
Introduction
In order to ensure the proper methodology and sound logic of scientific
studies, research results are often scrutinized by a panel of impartial
experts, a process called peer review. In recognition of the effectiveness
of this method, the Office of Management and Budget (OMB) issued a bulletin
that would establish government-wide peer review guidelines on regulatory
data used in the policy-making process. OMB claims the new guidelines will improve the quality
of scientific data used by federal agencies, and thus strengthen regulations from future assault. Critics argue
that the Bulletin is unnecessary and leaves scientific results vulnerable
to exploitation by those seeking to stall new regulations. OMB issued
two preliminary drafts of the proposal and received a number of comments.
The final version of the Bulletin was posted on December 15, 2004.
Background
In an attempt to standardize the way federal agencies utilize scientific
information, OMB's Office
of Information and Regulatory Affairs (OIRA)
drafted a series of guidelines dictating evaluation
standards for research data. The Proposed Bulletin
on Peer Review and Information Quality, which appeared in the
September 15, 2003 Federal Register, laid out peer review requirements for
all significant regulatory information disseminated by federal agencies.
While OMB cited several sources praising the benefits of peer review,1
critics of the Bulletin were quick to assert that no clear need for
reform had been established.2 OMB contends that the variability
in peer review policies
among federal agencies makes it difficult for the public to keep track
of important regulatory decisions.3 What OMB saw as a plan
to erase these inconsistencies in
regulatory standards, skeptics viewed as a possible attempt to stack review
commissions with industry scientists and inhibit meaningful regulations.4
OMB's authority to issue the proposal has also been called into question.
While the agency argues it is enforcing the Data Quality Act,
some groups claim that only Congress has the proper jurisdiction
and has chosen in the past not to implement new peer review policies.5
The Proposed Bulletin on Peer Review
The first OMB Bulletin attempted to distinguish between "significant
regulatory information" and "especially significant regulatory
information." For the former, OMB called for a mandatory peer review,
exempting data pertaining to national defense or foreign affairs. The
particular review methods were left to the agency's discretion.
For the latter, however, OMB laid out several rules to which agencies
must adhere. According to the Bulletin, regulatory information may be
deemed "especially significant" if it supports major regulatory
action, has an impact of more than $100 million a year, or is considered
of significant interagency interest by the agency administrator. Regulatory
information that falls into this category must undergo peer review subject to the following
rules:
- Reviewers must
a) have expertise in the given field
b) be independent of the agency under review
c) be free of real or perceived conflicts of interest.
- Peer reviewers must comment on science, not policy.
- Public Comment periods must be held, after which peer reviewers must
analyze the comments.
- The names and statements of all peer reviewers must be made public.
- Agencies may contract an outside firm to perform their peer review.
- Agencies must consult with the Office
of Science & Technology Policy (OSTP) and the OIRA concerning
their peer review guidelines.
- A waiver may be granted by the OIRA administrator (who may consult with
OSTP) for information pertaining to a health hazard, homeland security,
or other emergencies.
- Each year, agencies must provide to OIRA a list of scientific studies
effecting regulation and a plan for all peer reviews.
Reactions to the Bulletin
OMB received 187 comments on the
Proposed Bulletin.6
Some supporters, like the National Association of Manufacturers
(NAM), commended OMB, but called for broader applicability and more stringent
requirements.7 Many of the comments,
however, came from scientists and academic groups who claimed that a 'one
size fits all' approach is not suitable for the peer review process. Together
with several watchdog groups, they argued that the Bulletin had some potentially
harmful consequences that outweighed any good it might do.8
Several comments addressed the practicality of the measure. Mandatory
peer reviews require the increased compliance of qualified scientists.9
This would most likely consume great deals of
time and money, resulting in a slower regulatory process. Some even feared
an "endless cycle of review"10 and "paralysis by
analysis."11
In addition, many worried that the Bulletin would be open to exploitation,
since centralized peer review could be more easily manipulated.12
Public comment periods might be used to overwhelm the agencies
with responses and stall any regulatory actions.13
In addition, attempts to root out conflicts of interest could backfire,
since outside consultants are not bound by the
Federal Advisory Committee
Act (FACA) rules. This loophole was compounded by the lax
funding disclosure guidelines that were proposed.14
In the worst case scenario, agencies could be stripped of their capacity
to act quickly in response to an emergency.15
Some also feared that peer reviewers singled out by name might
feel compelled to keep their criticisms private.16
The most common objection, however, was that the Bulletin prevented the
most qualified scientists from participating in peer review.17
By focusing on biases that might arise between federally
funded scientists and the government, OMB neglected the possible financial
biases of industry scientists. Notably, the assumption that scientists
might be slanted towards their federal funding organizations has not been
supported by examples of misconduct.18
OMB Response to Public Comment
OMB issued a
Revised Bulletin on April 28, 2004, in which
they addressed many of the scientific community's concerns: the exclusion
of agency-funded scientists was altered; the prerequisite for "highly influential"
information was raised to $500 million; National Academy of Sciences
(NAS) standards for conflict of interest were adopted; and agencies were
given discretion to decide many peer review standards themselves. In addition,
agencies were instructed to post their peer review agendas online every
six months.
While scientific and academic groups were encouraged by the extensive changes,
many expressed frustration over the remaining exclusions on agency scientists
and questioned the need for the Bulletin in the first place.19
Final Draft of the Bulletin on Peer Review
After receiving 56 comments
on the second draft, OMB responded with a
third and final version
of the Bulletin on December 15, 2004. In many cases, OMB received contradictory suggestions from opposing
groups.20 OMB attempted to compromise, however,
often deferring to the policies of the NAS, as in the issue of financial
conflict of interest. The new Bulletin requires any financial ties between
a reviewer and the regulations under question to be examined. In a further
compromise, the identities, credentials, and comments of all reviewers
will be disclosed to the public, but names will not necessarily be linked
to particular statements.
In the final version, many of the lingering worries of scientific and academic
groups are carefully addressed. OMB encourages agencies to let public and professional
groups nominate peer reviewers, while allowing scientists who receive grants from the
sponsoring agency the chance to serve as well. Employees of the sponsoring agency still may
not serve as peer reviewers, but a rare exemption may be granted to scientists from
a different agency of the Cabinet-level department.
In order to ensure greater efficiency, the Bulletin
calls for specific time limits on public comment periods and leaves the
choice of peer review size and type to the discretion of the agency.
For monitoring purposes, the OMB requires all agencies to submit a final
report describing aspects of each peer review and public comment period.
In addition, proper documentation of peer review must be included in each
federal regulation.
As the Bulletin goes into effect next year, this brief will be updated
in order to document the impacts on scientific policy.
1 See page 5 of the
Revised Bulletin for OMB's explanation of
the need for Peer Review reform.
2 Comments from a letter
by several members of the House of Representatives: Henry Waxman (D-CA), John F. Tierney (D-MA), Sherrod Brown (D-OH),
Eddie Bernice Johnson (D-TX), Mark Udall (D-CO), Brian Baird (D-WA), and Michael M. Honda (D-CA).
3 OMB's explanation of the Bulletin
4 Waxman et al.
5
Objections to OMB's authority in the matter can be found in the
Comments from OMB Watch
6 OMB Comments Page
7
Comments from the National Association of Manufacturers (NAM)
8
Comments from the Federation of American Scientists (FAS)
9
Comments from the Council on Governmental Relations (COGR)
10 Waxman et al.
11
Comments from OMB Watch
12
Comments from OMB Watch
13
Comments from the Council on Governmental Relations (COGR)
14
Comments from the Federation of American Scientists (FAS)
15
Comments from the American Association for the Advancement of Science (AAAS)
16
Comments from the National Academy of Sciences (NAS)
17 Comments from
the National Academy of Sciences (NAS)
18 Comments from OMB Watch
19 Comments from
the Association of American Universities (AAU)
20
OMB's Response to Comments on the Second Draft
Updated January 24, 2005
|
|
