Competitive Proposal Review Options

Single-Stage Written Review

Independent Reviews

Each proposal will be assigned to three highly qualified reviewers selected for their expertise in the particular technical area of the proposal. The total number of reviewers to be recruited depends on the number of proposals received in the competition. Each reviewer will generate an evaluation of the assigned proposal(s). No travel is required for reviewers participating in the mail review round. All reviews will be included as the deliverable.

Use of multiple, independent reviews per proposal is useful when the goal of the peer review is to engage a breadth of expertise and generate multiple opinions that will be interpreted by a group able to compare and contrast the relative merits of the reviews. Because these reviews are completed independently, the focus of each critique and scoring and rating ranges may vary significantly.

Single Panel Review

When submitted proposals are similar in theme and few in number, it may be useful to engage a single review panel to evaluate the entire portfolio of proposals. The number of reviewers engaged will depend on the number of proposals submitted, variability in thematic elements, and review criteria, but will likely be between 3 and 5. In this scenario, each proposal will be assigned a primary, secondary, and tertiary reviewer, who will generate independent evaluations as above. When necessary, anticipated to be up to half of the cluster, an additional fourth review will be sought on specific questions. The fourth review will be used to inform the work of the other three reviewers. Following completion of individual reviews, reviewers will convene a conference moderated by AAAS staff to prepare a summary review for each of the proposals. The outcome of the process will be one summary evaluation report which expresses the consensus opinion of the review panel. Only the summary reviews will be included as the deliverable. Since the panel is able to discuss all proposals, variability in scoring and rating ranges is minimized. In addition, the panel is able to provide funding portfolio guidance if desired.

Single panel review is a good approach if proposals are based on limited preliminary data and requesting minimal (e.g., seed) funds. In this scenario, breadth of expertise, rather than depth of expertise in any one specific area is desired. However, when the number of submitted proposals increases, this approach may become impractical.

To minimize costs, the panel meeting can be accomplished by conference call moderated by AAAS staff. This will be a “virtual study section”.

Summary Reviews Generated by Multiple Panels

For certain competitions, submitted proposals may differ considerably in theme or are large in number and complicated by size or scope. In these scenarios, the proposals will be sorted into thematic clusters, such that the same three reviewers may consider multiple, similar proposals. In this approach, each “panel” of reviewers will provide a “single panel review” as described above. Again, the three reviewers will individually read the proposals and submit independent reviews. When necessary, anticipated to be up to half of the cluster, an additional fourth review will be sought on specific questions. The fourth review will be used to inform the work of the other three reviewers. Following completion of individual reviews, reviewers will convene by conference call moderated by AAAS staff to prepare a summary review for each of the proposals. Only the summary reviews will be included as the deliverable. Again, the outcome of the process will be one summary review per proposal that expresses the consensus opinion of the panel of reviewers.

Since each conference call will be moderated by AAAS staff, consistency in evaluations, scoring and ratings is maintained, though no one panel has seen the entire portfolio of submitted proposals. Because of this consistency, it is possible to compare proposals from different thematic clusters and rank, if appropriate.

Multiple-Stage Review

In competitions that require a significant investment by the funder, are complicated in scope, and/or involve components that are not easily evaluated solely through a written proposal, a multiple-stage review process may be desired. For multiple-stage reviews, an interview round is necessary. In this scenario, the first-stage of review will serve as a means to ensure that only the highest quality proposals advance to an interview. Any of the above single-stage written review processes may be used to generate this assurance. Those applicants invited to interview will receive their corresponding evaluation (or evaluations, as the case may be) from the written review round to aid in preparation. An interview panel will be convened in a convenient location and will consider up to 25 proposals. Interview panel members may be drawn from the pool of written reviewers, but will be specifically selected for appropriate expertise. Panel composition will be arranged to represent the spectrum of proposals to be interviewed. A senior AAAS staff member will facilitate the work of each panel.