Funding Research Randomly
In this post, Louis Larue (Aalborg University, Denmark) discusses his article recently published in the Journal of Applied Philosophy on the appropriateness of selecting research applications randomly.
Applying for external funding is an integral part of academic life. Universities dedicate huge amounts of resources, and often have entire teams of administrators and advisors, to help researchers obtain external grants and manage the immense load of paperwork required to administrate successful applications. Researchers and teachers, at all stages of their careers, spend considerable time and resources to write, read, revise, and submit applications. If successful, they will then have to write various reports and will be required to master the complex and often obscure language of funding agencies. At a more advanced stage of their careers, they will also dedicate a significant share of their time to reviewing and evaluating applications submitted by others and to sit in various selection committees.
Most of the time, the selection procedure involves (in one or several steps) the evaluation of the scientific quality of the submitted applications, by one or several peer reviewers. When all evaluations have been gathered, a selection committee usually selects successful applicants. The ideal behind this procedure (which I have only sketched, and which varies across countries and institutions) is to select, impartially, the “best” applications, that is, those with the highest level of scientific quality, properly defined.
Let’s call this selection procedure the “Peer Review procedure” (or PR). In recent years, it has attracted much criticism. For many, it is a costly, biased, and conservative procedure that is unable to deliver on its promise to select the best applications. In response to these criticisms, many authors have advocated mixed procedures involving various degrees of peer review and random selection (for instance, here and here). Following usage in the literature, I will call these mixed procedures “Modified Lotteries” (or ML).
The modified lottery is a two-stage procedure. At stage 1, the members of the selection committee select, among all eligible applications, the ones that they judge to be the most qualified applications, that is, those that meet minimal standards of scientific quality. At this stage, only the “worst” applications are rejected. The selection rate is thus allowed to be high, or, in any case, much higher than the current selection rate. At stage 2, a certain percentage of the applications selected at stage 1 is randomly selected. The percentage of applications selected at stage 2 is simply a function of the amount of money at the disposal of the funding agency.
In this post, I shall argue that the modified lottery procedure would strike a better balance between scientific quality, cost-effectiveness, impartiality, and fairness, than the current peer review procedure. (In the article, I also discuss, and reject, pure random selection, but I leave that part of the argument aside here).
Cost-effectiveness and scientific quality
A first intuitive argument for the use of random selection is that it would liberate time and money for researchers to do actual research. For the time dedicated to writing and reviewing applications amounts to time not dedicated to research and teaching. Considering the fact that most applications are rejected, this time is generally wasted.
However, the cost-reducing potential of random selection should not be over-estimated. A recent survey of applicants to the Health Research Council of New Zealand, which is among the first funders to use a Mixed Lottery, report that most of the applicants declared that they did not reduce the time spent writing their applications. Moreover, the time dedicated to reviewing proposals is not necessarily wasted. First, reviewers may be expected to set aside at least the proposals that do not meet minimal standards – an ability that should not be underestimated. Second, even if we assume that they cannot, getting rid of peer reviewers entirely may remove the incentive to write serious research proposals.
Hence, the relationship between the costs and benefits of investing time and money in selecting applications demands further consideration. In the article, I argue that costs are justified if they allow setting aside the applications that do not meet minimal standards of scientific quality; and that they are unjustified otherwise. Hence, dedicating time and money to peer reviewing applications is justified up to the point where peer reviewers can no longer perform their selection job. The empirical literature has for years stressed that peer reviewers are often unable to agree on the ranking of excellent applications, though they are more likely to agree on those applications that do not reach a minimal level of quality. The mixed lottery is thus to be preferred to the current system, because the limited space it gives to peer review allows to reduce its costs in a way that is not detrimental to scientific quality, since stage 1 is there to make sure that some peer reviewing still takes place. Though it may be impossible to find the “optimal” level of peer review, it is likely to be greater than zero and lower than the current level.
Impartiality and biases
A common complaint about peer review is that it is biased. There is evidence that the Peer Review procedure tends to be biased against women and ethnic minorities. Moreover, personal relationships as well as a preference for one’s own area of expertise tend to skew the peer reviewers’ evaluations. For all these reasons, a selection procedure based on peer review is unlikely to be impartial.
It is uncontroversial to say that these biases are bad, even morally wrong. Yet we may have reasons to accommodate some biases for the sake of retaining some place to peer review. In very short, my argument is the following: peer review is necessary to (at least) set aside the worst applications from the rest and to avoid removing the incentive of writing minimally good applications. Yet peer review is also inherently biased in some way. Hence, getting rid of all biases would require getting rid of peer review entirely, which would be detrimental to scientific quality. How do we get out of this dilemma?
My view is that, because peer review is unescapable, we should allow for the possibility that biases will influence the selection procedure. In that context, the modified lottery is preferrable to the current system, because it minimises the influence of biases, by leaving only a limited space to peer reviewers. However, those who would want to condemn biases more severely than I do, will have to contemplate the necessity to get rid of peer reviewers entirely and turn to pure random selection instead. My view is that the latter move would come at a cost for the advancement of science, because it would lower the probability to fund the best research. As I argue below, it may also be unfair.
Fairness
A further frequent complain against the current pee-review procedure is that it is unfair (see for instance here), though “unfair” is often confused with “biased”. However, this complaint may also be raised against proposals to select research proposals randomly (either partially or totally): isn’t that unfair to excellent applicants to consider all applications equally?
In the article, I use Broome’s idea that the fair distribution of a good requires that claims to a good should be satisfied in proportion to their strength. In our case, the good to be fairly distributed is research money. People’s claim to that good will depend on the extent to which their future research will be the most likely to produce the best science. Therefore, one may say that grants are distributed fairly when they are allocated to the proposals that have the strongest claim to research money, that is, to those that are the most likely to produce the best research.
In an ideal world without budget constraints, biases and other limitations, the peer review procedure would be the best and the fairest procedure: it would always select the most deserving applicants. But we do not live in such a world. First, in the real world, budget constraints may prevent funding bodies from giving money to all deserving applicants (i.e. those who have the strongest claim to it). Second, peer reviewers may be unable to reach a consensus on who the most deserving applicants are (a phenomenon that I call “epistemic limitations”). In that world, the modified lottery is the best choice.
As I have argued above, we may expect peer reviewers to be able to track scientific quality up to a certain point. If peer review has some value, the first stage of the modified lottery will allow to set aside the applications that have some minimal level merit (that is, a “minimal claim to research money”) from those who do not. The first stage therefore guarantees, at least to some extent, a certain degree of discrimination based on merit. But beyond that point, random selection is to be preferred, since no actual argument based on reasons may be used where epistemic uncertainty prevents reviewers from collectively distinguishing between applications. At stage 2, random selection ensures, at least, that all applications that have passed stage 1 have an equal chance to get funding, and that it is not biases or arbitrariness that decide among them.
The modified lottery is therefore not a fair procedure: it will not automatically distribute research money to those who have the strongest claim to it. But it is fairer than other procedures. It is fairer than pure random selection because it leaves some place to merit, which random selection fails to do; and it is fairer than the current system because, once the possibilities of peer review have been exhausted, it does not pretend to be able to select the best proposals among proposals whose relative merit is undistinguishable by reviewers (or disputed). Rather, it gives equal weight to all of them.
Some readers may still complain that the modified lottery disrespects excellent applicants, those who really deserve to be selected. In response, I would like to stress that the first stage of the proposal is meant to ensure that the best candidates are among the pool of short-listed applicants, and that they are selected according to shared standards of scientific quality. My view is that we cannot hope for more: it is beyond the capacity of peer reviewers to discover the “truly” best applicants. Moreover, the second stage limits the influence of non-scientific criteria (biases, etc), which might be present at stage 1, so that good candidates with profiles that are more likely to attract biases have a higher chance (compared to the present system) to be selected. So both stages actually contribute to increasing the ability of the procedure to track scientific excellence, rather than something else. Finally, we may have serious doubts that the current procedure is selecting the best applications. Lack of resources and various biases, as well as possible disagreements among evaluators on the quality of different applications, prevent the current system from doing its job well. Therefore, though there is a risk that the modified lottery will sometimes fail to select some of the best applications, this risk is probably not much higher than for the current peer review procedure.
Louis Larue is a researcher at the Aalborg University, Denmark; and a guest professor at the Hoover Chair of Social and Economic Ethics, UCLouvain, Belgium. He has published on the ethics of money and finance, and on several issues in the philosophy of economics. His first book, entitled Alternative Currencies: a Critical Approach, has just been published by Routledge.
