One attracts investment by the National Science Foundation — and they are no means alone, but decidedly dominant in science education — by engaging in one of the least altruistic, and most egotistical acts. It's called submitting a proposal for funding. This requires plans to help others by supporting oneself. It encourages hyperbolic promises and claims that appeal to reviewers' sense of complicity — since most of them are engaged in the same pursuit, and of honor — since they are expected to bring rigor and "objectivity" to the review.
Suffice it to say, reviewers tend to confuse others' proposed ideas as an invitation to advise sponsors on how they, the reviewers, would go about the task, instead of appraising the approach outlined. (Translation: reviewers have egos, too.) That reviewers are drawn from the same pool of competitors for scarce resources, there is an inevitable conflict of interest that feeds "peer review." If one is not sufficiently knowledgeable and skilled at articulating ideas relevant to a narrow band of research, then one is not valuable as a "peer." We are left with a dilemma of a research world divided in two: you are either a sponsor allocating a scarce resource or a performer competing for that resource on whom sponsors rely.*
My recent experience serving on review panels for the National Institutes of Health adds to this curious dynamic. As panels become increasingly interdisciplinary (see my "A FIRE Starter" blog post), granting benefit-of-the-doubt shrinks from view. Not quite a "gotcha" approach, but if, say, psychologists, education evaluators, biostatisticians, and life scientists comprise the review panel, the likelihood that a proposal will lack essential ingredients of theory, method, or analysis of outcomes that signal real effects of interventions is great indeed. And with five criteria rated on a nine-point scale (that's the NIH template), the potential for mischief looms. Does this process yield the most promising research on STEM education? Hard to tell. But it surely is a Rorschach test on what moves reviewers to wax glowingly or harshly on proposals (often submitted by multidisciplinary teams that bring comparable complexity to their task).
The window into the review process reveals the pathway to federal funding glory in the 21st century. The competition is fierce, with many bases to cover. The energy expended on the front end—preparing the proposal and reviewing it by sponsors' agents — is a substantial drain on scientists' time and creativity. Yet no one has yet devised a better system for allocating scarce resources. That makes this a lament without a solution.
But here's a thought: to complete the accountability loop, it would seem that the outcomes of funded research need to be included as a criterion for funding the next proposal. Otherwise, we remain either suckers for promises or egotists with grand claims. Like it or not, yea-sayers and naysayers exist and must co-exist in the current version of funding the pathway.
- For an analysis of the various aspects of the dilemma, see D.E. Chubin and E.J. Hackett, Peerless Science: Peer Review and U.S. Science Policy. SUNY Press, 1990.