AAAS Policy Brief: OMB Bulletin on Peer Review

Issue Summary | Resources

---

Contents

• Introduction
• Background
• The Proposed Bulletin on Peer Review
• Reactions to the Bulletin
• OMB Response to Public Comment
• Final Draft of the Bulletin on Peer Review

Introduction

In order to ensure use of appropriate methodologies and sound logic of scientific studies, research results are often scrutinized by a panel of impartial experts in an internatinally recognized and practiced process called peer review. The Office of Management and Budget (OMB) issued a bulletin that would establish government-wide peer review guidelines on regulatory data used in the policy-making process. OMB claims the new guidelines will improve the quality of scientific data used by federal agencies, and thus strengthen regulations from future assault. Critics argue that the Bulletin is unnecessary and leaves scientific results vulnerable to exploitation by those seeking to stall new regulations. OMB issued two preliminary drafts of the proposal and received a number of comments. The final version of the Bulletin was posted on December 15, 2004.

Back to top

Background

In an attempt to standardize the way federal agencies gathers scientific information, OMB's Office of Information and Regulatory Affairs (OIRA) drafted a series of guidelines dictating evaluation standards for research data. The Proposed Bulletin on Peer Review and Information Quality, which appeared in the September 15, 2003 Federal Register, laid out peer review requirements for all significant regulatory information disseminated by federal agencies.

While OMB cited several sources praising the benefits of peer review,¹ critics of the Bulletin were quick to assert that no clear need for reform had been established.² OMB contends that the variability in peer review policies among federal agencies makes it difficult for the public to keep track of important regulatory decisions.³ What OMB saw as a plan to erase these inconsistencies in regulatory standards, skeptics viewed as a possible attempt to stack review commissions with industry scientists and inhibit meaningful regulations.⁴

OMB's authority to issue the proposal has also been called into question. While the agency argues it is enforcing the Data Quality Act, some groups claim that only Congress has the proper jurisdiction and has chosen in the past not to implement new peer review policies.⁵

Back to top

The Proposed Bulletin on Peer Review

The first OMB Bulletin attempted to distinguish between "significant regulatory information" and "especially significant regulatory information." For the former, OMB called for a mandatory peer review, exempting data pertaining to national defense or foreign affairs. The particular review methods were left to the agency's discretion.

For the latter, however, OMB laid out several rules to which agencies must adhere. According to the Bulletin, regulatory information may be deemed "especially significant" if it supports major regulatory action, has an impact of more than $100 million a year, or is considered of significant interagency interest by the agency administrator. Regulatory information that falls into this category must undergo peer review subject to the following rules:

• Reviewers must
     a) have expertise in the given field
     b) be independent of the agency under review
     c) be free of real or perceived conflicts of interest.
  
• Peer reviewers must comment on science, not policy
• Public Comment periods must be held, after which peer reviewers must analyze the comments
  
• The names and statements of all peer reviewers must be made public
• Agencies may contract an outside firm to perform their peer review
• Agencies must consult with the Office of Science & Technology Policy (OSTP) and the OIRA concerning their peer review guidelines
• A waiver may be granted by the OIRA administrator (who may consult with OSTP) for information pertaining to a health hazard, homeland security, or other emergencies
• Each year, agencies must provide to OIRA a list of scientific studies effecting regulation and a plan for all peer reviews

Back to top

Reactions to the Bulletin

OMB received 187 comments on the Proposed Bulletin.⁶ Some supporters, like the National Association of Manufacturers (NAM), commended OMB, but called for broader applicability and more stringent requirements.⁷ Many of the comments, however, came from scientists and academic groups who claimed that a 'one size fits all' approach is not suitable for the peer review process. Together with several watchdog groups, they argued that the Bulletin had some potentially harmful consequences that outweighed any good it might do.⁸

Several comments addressed the practicality of the measure. Mandatory peer reviews require the increased compliance of qualified scientists.⁹ This would most likely consume great deals of time and money, resulting in a slower regulatory process. Some even feared an "endless cycle of review"¹⁰ and "paralysis by analysis."¹¹

In addition, many worried that the Bulletin would be open to exploitation, since centralized peer review could be more easily manipulated.¹² Public comment periods might be used to overwhelm the agencies with responses and stall any regulatory actions.¹³ In addition, attempts to root out conflicts of interest could backfire, since outside consultants are not bound by the Federal Advisory Committee Act (FACA) rules. This loophole was compounded by the lax funding disclosure guidelines that were proposed.¹⁴

In the worst case scenario, agencies could be stripped of their capacity to act quickly in response to an emergency.¹⁵ Some also feared that peer reviewers singled out by name might feel compelled to keep their criticisms private.¹⁶

The most common objection, however, was that the Bulletin prevented the most qualified scientists from participating in peer review.¹⁷ By focusing on biases that might arise between federally funded scientists and the government, OMB neglected the possible financial biases of industry scientists. Notably, the assumption that scientists might be slanted towards their federal funding organizations has not been supported by examples of misconduct.¹⁸

Back to top

OMB Response to Public Comment

OMB issued a Revised Bulletin on April 28, 2004, in which they addressed many of the scientific community's concerns: the exclusion of agency-funded scientists was altered; the prerequisite for "highly influential" information was raised to $500 million; National Academy of Sciences (NAS) standards for conflict of interest were adopted; and agencies were given discretion to decide many peer review standards themselves. In addition, agencies were instructed to post their peer review agendas online every six months.

While scientific and academic groups were encouraged by the extensive changes, many expressed frustration over the remaining exclusions on agency scientists and questioned the need for the Bulletin in the first place.¹⁹

Back to top

Final Draft of the Bulletin on Peer Review

After receiving 56 comments on the second draft, OMB responded with a third and final version of the Bulletin on December 15, 2004. In many cases, OMB received contradictory suggestions from opposing groups.²⁰ OMB attempted to compromise, often deferring to the policies of the NAS, as in the issue of financial conflict of interest. The new Bulletin requires any financial ties between a reviewer and the regulations under question to be examined. In a further compromise, the identities, credentials, and comments of all reviewers will be disclosed to the public, but names will not necessarily be linked to particular statements.

In the final version, many of the lingering worries of scientific and academic groups are carefully addressed. OMB encourages agencies to let public and professional groups nominate peer reviewers, while allowing scientists who receive grants from the sponsoring agency the chance to serve as well. Employees of the sponsoring agency still may not serve as peer reviewers, but a rare exemption may be granted to scientists from a different agency of the Cabinet-level department.

In order to ensure efficiency, the Bulletin calls for specific time limits on public comment periods and leaves the choice of peer review size and type to the discretion of the agency.

For monitoring purposes, the OMB requires all agencies to submit a final report describing aspects of each peer review and public comment period. In addition, proper documentation of peer review must be included in each federal regulation.

Back to top

¹ See page 5 of the Revised Bulletin for OMB's explanation of the need for Peer Review reform.
² Comments from a letter by several members of the House of Representatives: Henry Waxman (D-CA), John F. Tierney (D-MA), Sherrod Brown (D-OH), Eddie Bernice Johnson (D-TX), Mark Udall (D-CO), Brian Baird (D-WA), and Michael M. Honda (D-CA).
³ OMB's explanation of the Bulletin
⁴ Waxman et al.
⁵ Objections to OMB's authority in the matter can be found in the Comments from OMB Watch
⁶ OMB Comments Page
⁷ Comments from the National Association of Manufacturers (NAM)
⁸ Comments from the Federation of American Scientists (FAS)
⁹ Comments from the Council on Governmental Relations (COGR)
¹⁰ Waxman et al.
¹¹ Comments from OMB Watch
¹² Comments from OMB Watch
¹³ Comments from the Council on Governmental Relations (COGR)
¹⁴ Comments from the Federation of American Scientists (FAS)
¹⁵ Comments from the American Association for the Advancement of Science (AAAS)
¹⁶ Comments from the National Academy of Sciences (NAS)
¹⁷ Comments from the National Academy of Sciences (NAS)
¹⁸ Comments from OMB Watch
¹⁹ Comments from the Association of American Universities (AAU)
²⁰ OMB's Response to Comments on the Second Draft

Updated January 24, 2005