An alternative procedure for allocating research grants

This is a guest post by Louis Larue, Marie Skłodowska-Curie Postdoctoral Fellow at Aalborg University.

Applying for external funding is an integral part of academic life. Universities dedicate huge amounts of resources, and often have entire teams of administrators and advisors, to help researchers obtain external grants and manage the immense load of paperwork required to administrate successful applications. Researchers and teachers, at all stages of their careers, spend considerable time and resources to write, read, revise, and submit applications. If successful, they will then have to write various reports and will be required to master the complex and often obscure language of funding agencies. At a more advanced stage of their careers, they will also dedicate a significant share of their time to reviewing and evaluating applications submitted by others and to sit in various selection committees.

In general, the evaluation procedure involves (in one or several steps) the evaluation of the scientific quality of the submitted application, by one or several peers. When all evaluations have been gathered, a selection committee usually selects successful applicants. The ideal behind this procedure (which I have only sketched here and which varies across countries and institutions) is to select, impartially, the “best” applications, that is, those with the highest level of scientific quality, properly defined.

I do not deny the value of this ideal, but it is far from realized in practice. The reform proposal that I defend below is meant to reinvigorate this ideal and salvage it from several threats.

The limits of the current model

First, in the real academic world, people face time and resource constraints. Most of the work described above is unpaid work (some reviewers and committee members are paid, though) and the time dedicated to these tasks amounts to time not dedicated to research and teaching. Taking into account the fact that most applications are rejected, this time is generally wasted (even if applications can be recycled into new applications, I take it that the possibility of recycling does not compensate for the time lost writing, reading, commenting, revising, submitting, reviewing and evaluating unsuccessful applications). Even if the actual gains of applying for external funding for individual researchers and their institutions may still exceed the costs, the current system can still be said to suffer from severe ineffectiveness.

Beyond these efficiency considerations, my own experience of several years of applying in various European countries is that the application process is always extremely long, especially in comparison with non-academic sectors (never shorter than 6 months between submission and final response, in my experience) and obscure (no or very parsimonious feedback, irrespective of whether the application is successful or not), which may give the impression that the whole process is just a lottery, as one former colleague once told me.

In addition, we may have serious doubts about the impartiality of the system. Many researchers are, simultaneously or at different points in their careers, both applicants and reviewers. This situation opens the door for serious conflicts of interest, as well as for the exchange of favors (I push your application this time, expecting you will do the same for me in the future). Moreover, reviewers, even the most honest and rigorous ones, may have a tendency to favor projects that adhere to their own methodological or theoretical preferences. In that case, the selection process will not track scientific excellence but the reviewers’ interests or preferences. Finally, selection committee cannot be thought to be exempt from implicit and explicit biases against various groups or individuals, on the basis of traits that should be irrelevant for the selection of applications (gender, sex, nationality, ethnicity, etc).

The alternative proposal

In this post, I propose a way to improve the current grant selection process (which I take to be broadly equivalent in most European countries, at least in those where I have myself applied) by adding to it a random selection device. The purpose of my proposal is to strike the right balance between impartiality (roughly, the absence of biases and conflicts of interest) and scientific quality (choose your favorite definition). As government funding for higher education is being cut (e.g. in Denmark) and as more and more PhD students are seeking a way to prolong their stay in academia, reliance on external funding increases. This makes the task of finding an appropriate way to process applications even more urgent.

I propose to proceed in two stages.

At stage 1, the members of the selection committee select, among all eligible applications, the ones that they judge to be the most qualified applications, according to the criteria of excellence specified by the scheme for which they are evaluating applications and informed by the reports of the external reviewers. The key point is that the number of applications selected at this stage is not constrained by budget considerations but only by consideration of scientific excellence. It is thus allowed to be high, or, in any case, much higher than the current selection rate. Depending on the number of submitted applications, a maximum may or may not be chosen. Yet, if there is a maximum, it should be high (say 25%). Let me call x the number of selected applications at stage 1.

At stage 2, y% of these x applications are randomly selected. The percentage of applications selected at stage 2 (y) is simply a function of the amount of money available (a complication may arise if different applications to the same funder require different amounts of money. For the sake of simplicity, I ignore this complication here and assume that all submitted applications would require the same amount of money).

I think this procedure would have several advantages, while still satisfying the ideal of scientific quality.

First, using random selection would reduce the costs (in time and money) of selecting research projects. The costs would be reduced for all people involved: it would shorten the waiting time for applicants, the time required to sit in meetings for evaluators and reviewers, and the administrative costs of handling the whole process.

Second, using random selection would also reduce the extent to which biases, conflicts of interest, and the exchange of favors, influence the current selection process.

Third, the fact that selection at stage 1 depends only on scientific considerations (and not on budget constraints) would reduce the risk to miss good applicants. In a similar vein, random selection at stage 2 will reduce the difficulty and the arbitrariness involved in choosing among excellent applicants, where it is impossible to choose all of them, or difficult to rank them.

Fourth, making explicit that the selection process is partly a lottery may reduce (a bit) the (sometimes) excessive pride of those who have been successful. It may also reassure the unsuccessful applicants by clarifying the reasons for rejecting their applications: either it was insufficiently good (if rejected at stage 1) or it was good but rejected for budget reasons (if rejected at stage 2).

The details of the proposal certainly need to be refined (I am for instance unsure about the appropriate percentage of applications to be selected at stage 1). But, overall, I think that this two-step procedure can strike a better balance between scientific quality and impartiality than the current one. In short, the first step ensures that applications are chosen according to shared standards of scientific quality. The second step increases the impartiality of the procedure, as compared to the current procedure.

Anticipating some objections

I would like to conclude this post by considering two objections: the first is that the use of a random device may fail to select the best applications, and even be disrespectful to the best candidates, who deserve to be chosen; the second is that it does not meet minimal standards of scientific quality.

In response to the first objection, I would like to stress once again that the first stage of the proposal is meant to ensure that the best candidates are among the pool of short-listed applicants, and that they are selected according to shared standards of scientific quality. Moreover, the second stage limits the influence of non-scientific criteria (biases, etc), which might be present at stage 1, so that good candidates with profiles that are more likely to attract biases have a higher chance (compared to the present system) to be selected. So both stages actually contribute to increasing the ability of the procedure to track scientific excellence, rather than something else. Finally, as I argued in the first part of this port, we may have serious doubts that the current procedure is selecting the best applications. Lack of resources and various biases, as well as possible disagreements among evaluators on the quality of different applications, prevent the current system from doing its job well.

A second possible objection is that my proposal pays insufficient respect to minimal standards of scientific quality. In response, let me make clear that, if human beings had no tendency to be influenced by implicit and explicit biases and had infinite resources to dedicate to the selection of the best applications according to clear standards that would be agreed upon by the entire scientific community, then, certainly, my proposal would be a “less scientific” second-best. However, we have reasonable grounds to believe that the current way in which applications are selected (at least, in countries where I have applied myself, that is, in most of Europe) is far from satisfying these conditions. The lack of time and resources required at all stages of the process, its obscurity, and the many biases that plague it greatly weaken the ability of the current procedure to satisfy high standards of scientific quality. My proposal is a second-best that is preferable (I think) to the current third-best.

Louis Larue

Warm thanks to Pierre-Étienne Vandamme for his excellent comments and for the invitation to publish this piece on this blog.

Louis Larue is a Marie Skłodowska-Curie Postdoctoral Fellow at Aalborg University, Denmark. His research focuses on issues of justice in money and finance, and on several topics in the philosophy of economics and social ontology. Besides philosophy, he is also a classical pianist and a church musician.