In recent years, the National Institutes of Health has made increasing use of targeted research for specific diseases or topics, including Requests for Applications (RFAs) and other grant mechanisms (see graph at right). NIH uses these mechanisms to encourage research in particular areas of public interest, or to open up new fields of science. They're an alternative to the more traditional investigator-initiated "open" mechanisms.
But how effective are RFAs at generating new science? How do scientists respond? And how much does it cost to get scientists to switch fields?
A new paper by Kyle Myers, an economist at Harvard Business School, attempts to answer these questions using NIH application and award data, and scientists' publication records from PubMed. The paper is forthcoming from the American Economic Journal: Applied Economics; a working paper version is available on Myers' website (PDF). Check out the paper for the full methodology, but here are the main findings:
- Scientists tend to pursue RFAs that are similar to the work they're already doing.
- There's an "RFA premium" to induce scientists to change focus: a typical RFA has to be 65% larger than a regular "open" competition to get scientists to change directions to pursue it, given "adjustment" costs (see graph).
- RFAs are indeed effective at stimulating new scientific publications in their target areas, though the effect is temporary. After a few years, scientists tend to revert to their original areas of study.
- Even with the "RFA premium," given the other factors above, RFAs appear to be a more cost effective means of generating new publications on a per-paper basis.
AAAS recently got to talk to Myers about the paper, the findings, and the implications. The below Q&A has been edited for length and clarity (the graphs on this page are also courtesy of Myers).
What was your motivation for choosing this as a research topic?
I started as someone who was interested in the economics of science more generally, and had spent some time in biomedical laboratories where I watched Principal Investigators make these kinds of decisions. I found myself starting to think about how scientists make this choice. I had done some internships at the NIH in my earlier years of schooling, so I was always intrigued by that institution just because it was such a large, powerful player in the science policy and science funding arenas. I was just really trying to understand how they manage that allocation process of who should get money to do what, and how much should we leave scientists to their own devices versus trying to nudge them in certain directions. Naturally, a way that we try to incentivize researchers is by allocating resources to particular [research topics]. So, my own research question became: how much does it cost to get scientists to change what they're working on?
The picture in my head is of scientists on a beach hunting for treasure. And my question for this paper was, “how thick is the sand?” If you're the government and you want to get scientists to move around, you can drop treasure in certain places [and hope that scientists will move towards it]. But if that sand’s really thick, then it will take larger treasures to get scientists to move, which means you [will have to] really want them to move, to try to trudge through that sand. And so that's the metaphor. That got me thinking, “Can I measure the treasure, or can we measure how thick the sand is and how far they have to walk in it?” Because it's important that we understand these things so we can think about how much “treasure” we (the policymakers) drop in certain places.
And when you do what you need to do, econometrically speaking, [you find that scientists] like to pursue research topics that are closer to them, and they like to pursue bigger pots of money. After accounting for things like competition, comparing the magnitudes of these two forces gives you an estimate of what I call “the elasticity of science,” what is effectively the thickness of the sand based on how willing they are to chase bigger pots over longer distances.
The findings seem rather difficult for RFAs. You find about a 65% premium for RFAs. In other words, the pot of treasure, as you call it, has to be 65% larger to get a scientist to chase it if the research topic is quite different from what they’re currently doing. Do you have any thoughts about what is underlying this cost premium?
There’re definitely tangible costs. If you walk into any modern laboratory, especially in the biomedical sciences, you're going to see a lot of pieces of equipment that cost tens or hundreds of thousands of dollars. So, there's going to be capital costs. There's going to be labor costs, right? No scientist these days is working by themselves, particularly in the biomedical arena, and so if a scientist wants to move into a new research area, maybe that's an area that they don't have any experience with the equipment or tools or methodologies that are needed and they might have to hire a postdoc or a research assistant.
And then there's a lot of research showing that scientists have very strong preferences over their work, and there's evidence that scientists are willing to take pay cuts just so they can have jobs that allow them to publish. And so, it's very easy to have a story where even if you ignored some of these capital costs, scientists would be reluctant to change their work. Scientists spend a long time learning particular niches of science and want to keep working on those topics. Sometimes it's a puzzle that you just really enjoy tinkering with, sometimes you really think that that is the most important thing that someone in your shoes could be working on.
And then there's certainly bound to be plenty of social interactions and peer pressures that are forcing people into or out of certain topics. One of the more interesting economics of science papers of late on this by Pierre Azoulay, Christian Fons-Rosen, and Josh Graff Zivin is called, “Does Science Advance One Funeral at a Time?” There are a lot of things that can push scientists to stay in their lane.
On the output side, you find RFAs do result in increased publications in their target areas, which suggests they’re successful in stimulating new science, but this effect is only temporary. After a few years, scientists tend to return to what they were working on previously. For society or taxpayers, what’s it worth to get this bump in publications output for a few years?
There are reasons it can be good that it's costly for scientists to move around. One of them is that the way science works is that it tends to reward priority. You can think of science as a big contest where everyone's trying to find the next big thing in certain places. But, one of the problems that can arise when people participate in contests is that they tend to race and “over-invest,” and much too many people will enter into these contests. And so, it might be a good thing that everyone stays where they start, because otherwise we might just have a bunch of scientists going over here and then over there, and that might not be optimal from a dynamic standpoint. We may not want scientists just hopping from lamppost to lamppost, trying to solve the next big puzzle.
Thinking about the value of these publications, we don't have a lot of good evidence yet as to how to think about putting any kind of dollar value on what these publications are worth. Some researchers are starting to get there. [Azoulay, Zivin, Li, and Sampat] have a great paper on the NIH where they start to get at a return-on-investment, but the lag between these publications and anything that we can put a dollar value on is so long. [Quantifying these returns] is the frontier in our field right now.
Given the costs of redirecting scientists, and your finding that the effect on publications is positive but temporary, does that cast some negative light on the idea of trying to direct research efforts?
The people that choose to work on these RFAs, and those that win them, are not a random sample of scientists. They’re purposeful in their choice. My results suggest that scientists do have a good sense of which one of them would be the best go after an [RFA]. If you think about it on a dollar per publication basis, the RFAs are actually more effective in the sense that it's cheaper to get a publication, because it appears that scientists self-select on their own knowledge that they have a good chance of doing well in that competition for the grant and producing publications.
The magnitude of these adjustment costs [for scientists changing direction] do make you think, are there cheaper ways that we might be able to help scientists move around if they want to? Because again, like the self-selection effects suggest, if they're given the opportunity and the extra resources to make that move, they're more productive. Under current policies, it would be very hard to get that time and money from [funding agencies] because most of these systems are built around what people refer to as “project- style” grants, where you get money to do science, to produce publications.
But I think there's some good evidence already that shifting to what they call “people-style” grants, where much more discretion about what the funds are used for is left up to the scientist, can have a lot of beneficial effects. And part of those effects might be due to the fact that they can take some time or they can use some resources to learn new things or to adjust and experiment before they get to the point of having to do their actual experiment and figure out what's a good new avenue for their research. That kind of shift towards more flexibility with what you can use those research funds for is definitely an interesting thing to think about. Now, of course, that opens up classic agency problems where we might start to worry about scientists using those funds efficiently because, you know, humans are human. But that's an area where I think we could push a little bit further into seeing what those tradeoffs are like given our current position.
It would be very interesting to see what would happen if an RFA, instead of being a traditional R01 mechanism, was an R01 amount of money just to start working on a topic in whatever way that you see fit. Now that opens up a host of new issues. My sense is that it certainly is easier to evaluate projects than a person's entire potential. And so, again, these are not easy tradeoffs to make, but it seems like there are certainly other ways of facilitating people of entering these new research areas.
So let's talk a little bit about the National Science Foundation [NSF] and the Endless Frontier Act. A goal of this bill is to establish a large new technology directorate to fund research in priority areas. Some of that research funding would flow through the traditional NSF directorates. Your findings seem to imply that we should be considering the elevated costs for redirecting some scientists toward these fields, although you also make the point that for scientists who are already working in these areas, it may actually enhance productivity.
The way that I would start to think about it through the lens of my paper would be, if this is part of a big shift, and we want more people working on [these technology areas] -- which is way outside my scope of knowledge to say anything about that -- but if we take that as true, then I would ask: how do we pay all of the fixed costs that we need to as soon as possible?
We would expect that eventually, someone who is entering high school or college, as they see this kind of bill being enacted, might think “Hmm, maybe that is a field where there's lots of resources for me to make an important impact.” But those people won't be entering the workforce for five, six, ten years. In the meantime, a lot of the work will be done by people who are already there [in the target fields], or who are not there but are experienced enough to consider entering.
And so, my first thought would be to try to alleviate the adjustment costs for people to enter in the early years as a part of this moonshot until we can get into a new equilibrium where people in the academic pipeline are choosing these fields at a higher rate. Eventually, we probably won't need to keep getting the computational biologists to come over and work on something that's more in line with the objectives here, because there [will be] people coming through Ph.D. and masters and even undergrad programs that are ready to make contributions. But like I said, that's probably going to be a while.
Do we know much about that students making choices of study or career based on different funding changes? Obviously, we had the NIH doubling 20 years ago, which resulted in a huge influx of new PhDs.
So the two things that come to mind are, first, with respect to the doubling, Richard Freeman and John van Reenen have a nice piece that tried to diagnose what happened during the NIH doubling. One of the things their paper makes the case for is: whatever you do, be smooth. In the sense of the NIH doubling, it looks like a lot of what it did was create an influx of foreign postdocs because they could quickly respond to the extra resources. As you probably know, dollars will get spent, whether they are really needed or not, and so if you jam money down scientists’ throats, they're going to find people to hire. They're going to find machines to buy. But if you're a bit smoother in the way you phase in these kinds of investments, then maybe it can have effects on individuals in high school and college, and by the time the funding is ramping up there's a larger supply of the skilled labor to make good use of those dollars.
The other thing that comes to mind is work that tries to understand why certain people become scientists. Pian Shu has done great work to diagnose how undergrads choose between STEM degrees versus finance degrees, since there are sometimes worries that finance “steals” people who would have made great scientists. She doesn’t find evidence of that, which is a good sign. But on the other hand, there is good evidence that there are many individuals, particularly among women, minorities, and low-income households, that did not pursue careers as inventors because of a lack of exposure to these kinds of careers in childhood. So, we certainly still have a lot of work to do to make sure that people with great potential are given opportunities to pursue science and invention, regardless of where they come from, or what they look like.
Most of the areas prioritized in the NSF bill are heavy on physical science, computer science, engineering, while your paper focuses on life sciences. Is there any reason we might expect your findings to apply differently to these other disciplines?
If it was the ‘80s and you said “laboratory,” I think people would think of a bench science, biochem-type lab with lots of equipment and lots of people. And you still see that a lot in the biomedical sciences, but these days, when you say “laboratory,” you very easily could be referring to a mathematician who has a lot of students working with them or a computer scientist who has a tremendous amount of equipment that they're using to do their work. And so, from the sorts of tangible costs that we talked about, there's very good reason to think adjustment costs are important in these fields.
And on the intangible cost side...No field is immune to all of the preferences and norms and social forces that are probably driving scientists’ decisions about what they're choosing to work on. I wouldn't expect it to be particularly easy for any scientist in any field to think about making dramatic changes to their agenda. Because a five- or six-year PhD really gets your brain set in a way that you, for good reasons and maybe bad reasons, you really think that what you're doing is the most important thing that you can be doing.
So, I think it is likely that the costs are probably quite large in the fields this bill is focusing on, but that's something we don't exactly know. I think one of the possibilities for [a new NSF directorate] would be to explore how large are these kinds of costs, what are ways that we can help researchers find the best project for them and society. This feels like an interesting opportunity where some exploration or even experimentation of different ways of getting resources to different types of scientists could be helpful in the long run, so that we have a better sense of how scientists respond and how they can best be supported.