In Praise of the Disappearing Art of In-Person Peer Review

Scientific research pre-1950 was a far smaller and less organized effort than what is currently offered. As the enormous value of investment in scientific research was realised post-WWII, the amount of money proffered to support research ballooned. Coupled with a significant expansion in post-secondary education with the building of new and enhancements of existing universities, an ecosystem for scientific and other forms of research blossomed, along with the training of a new generation of researchers. But with the increasing expenditures of tax payer funds there was a need to demonstrate accountability and fiscal responsibility. But how best to judge whether somebody’s idea about something very few people had even heard of was more compelling of support than anothers idea? Hence, was born peer review: review of proposals of scientists by scientists. The concept of expert evaluation also extended to scientific journals as these had become the “currency” of scientific production. If an experimental study was conducted, it’s value or likelihood of being correct/useful/appropriately robust could be judged by other scientists, who were engaged (for free) by the journal and remained anonymous to the submitting researchers.

Many researchers today see peer review as a necessary quality control of scientific research (footnote 1). There are now far more ideas and studies that can be funded or deserve wide attention when published. Success rates of grant applications typically hover between 8–20%, meaning the vast majority of submissions are rejected, despite the intense effort needed to assemble and support them. Though disheartening, few scientists would prefer there to be no review — afterall, there wouldn’t be any increase in funding with elimination of peer review. But this compact between scientists and the punishingly difficult peer review process rests on the confidence of the scientists in the fairness and effectiveness of said process.

Indeed, there have been many discussions about reforming or improving peer review — both of grant applications and publications — the latter having been distorted into a high-profit margin activity exploited by large publishing houses that charge the submitting scientists substantial fees and yet rely on the free service of their peers to actually conduct the peer review. But journal review is another topic for another day. Arguably, in a world of AI tools, broad internet sharing, near instant publication of pre-prints and the existence of freely accessible repositories (e.g. arXiv, BioRxiv and MedRiv, among many), it shouldn’t matter where a study is “published”, the cream should float to the top over time.

In any case, it is not as though there haven’t been major changes in grant peer review — the problem is, many of these have been detrimental. Canadian health researchers had a taste of major change when their major federal funding agency rolled out major reforms of programs. These included dispension of peer review panels (groups of scientists working in a particular field who come together to rate grant applications over a 2–3 day period) in favor of asynchronous virtual review. Traditional peer review assigned each expert reviewer 8–10 applications for them to assess and each grant had 2 or 3 reviews. The argument was that if 5 reviewers were assigned, even if they were not in the same room and didn’t interact, then statistically, their overall ranking would be more accurate, less subject to individual bias and therefore lead to a better and fairer outcome. Theoretically, that is, as the new process failed to account for human behaviour. By removing the need to explain and justify a review and ranking to other scientists in the room, the amount of effort and care put into a review inevitably dropped (there is a lot more detail here: https://medium.com/@jwoodgett/one-step-forward-towards-a-precipice-a464a641b7ab and a follow up here: https://medium.com/@jwoodgett/peer-review-and-the-real-cost-of-popularism-or-how-to-get-640-comments-on-an-article-d9522d4cd668 ). The result was wildly incongruent rankings and a break in confidence of both the reviewers and the scientists applying (footnote 2). Thanks to a campaign and the support of the then Minister of Health, it was soon scrapped and the old system reinstated.

That was until the pandemic when it simply wasn’t feasible to gather researchers in a gloomy hotel basement meeting room in Ottawa or Bethesda. And so, the in-person panels moved on-line (in the case of Canadian agencies, using the AMC Gremlin of video conferencing software, MS Teams). This came with some other benefits. Reviewers on the West Coast no longer had to lose an additional travel day (not to mention the associated carbon footprint), there were flight and hotel savings and those with young kids didn’t need to arrange child care. Moreover, the review discussion was done synchronously, a big difference from prior experiments with asynchronous reviews. These benefits are substantive and in the case of several funding agencies were sufficient to retain on-line reviewing after the pandemic had been officially declared over (sad trombone).

So, is on-line reviewing as good, better or worse than in-person reviewing? If as good or better, then, with the other benefits, it’s a shoo-in. But has this been evaluated? This is not a trivial question to answer because it requires assessment of the quality of grants that are funded. You could use a dart board to adjudicate those grants that meet a given success rate, or ask JD Vance to select a few from the sofa cushions and pick 20% of submitted grants. Whether peer review is effective or not requires determination of whether the best grants were funded — and that is aways going to be subject to some subjectivity.

Until an agency does a head-head comparison (requiring a duplication of effort, several times), here are some observations and considerations:

1. Scientists are busy. They must find time to read and write reviews and then participate in a review panel. We are all distracted and will do other work when we can. This typically doesn’t (or is less likely to) happen at an in-person panel as you don’t know who is looking at you and there is an expectation that you are focused on the task at hand. For many on-line panels the majority of participants not only have microphones muted but also their cameras off at any given time.
2. Because you are under the microscope of other colleagues, a reviewer must be prepared to defend their review. Your assessment is being heard by 10–15 other experts and no one wants to look or say something that makes them look foolish. Reputations are made at such panels. Reviewer astuteness, fairness, and objectivity are respected and appreciated as a skill.
3. The Chair of the panel is charged with conducting a fair review process. With people in the same room, there is greater influence of the Chair and greater self-awareness of reviewers.
4. Discussion is often vital. Especially by panel members who were not assigned a particular application. They are acting as jurors, observing the conduct of the assigned reviewers and can often spot inconsistencies and raise them. When the assigned reviewers are close in score/enthusiasm, the overall score is typically close to the mean of the declared reviewers. However, if initial scores are divergent, the panel is tasked with assessing who has most fairly judged the application. That requires interactivity.
5. Everyone on a panel has at least some expertise for every grant being evaluated (that’s why you are there). Discrepant information, overstatements, incorrect assumptions, omissions, etc. can be missed by the assigned reviewers but can be picked up by attentive panel members, if there is discussion.
6. Participating on a panel requires time and effort and this is offset by service obligation and new knowledge learned from reading other well constructed grants. But a reviewer only gets out what they put in and if distracted, or allowing themselves to be, will feel less affinity and buy-in for the process.
7. While secondary, there is a social aspect to meeting with other researchers. It’s why conferences exist. It builds commeraderie and also provides direct interaction with the grant funding personnel. How many collaborations were sparked by a panel dinner conversation? How many early career researchers made new connections?
8. On-line reviewers are more dettached from the applications and less invested in their outcome. This is somewhat akin to the Milgram experiment where participants were instructed to press a button that they were told gave another person an electric shock. By distancing their actions from the consequences, they were more harsh/cruel than expected.
9. Related to the above, the percentage of grants that are not discussed has edged up in many competitions (e.g. NIH RO1 and CIHR Project Grants). Often greater than 50% do not benefit from even scant discussion. This certainly streamlines the process and increases efficiency and use of time but while other reviewers can “rescue” a streamlined grant for discussion, this is relatively rare. The submitting scientist does get a pair of reviewer reports but no context. Moreover, the content of these reports is not heard by the rest of the panel and errors, etc. can slip through — although these reports are meant to be checked for inappropriate statements by the funding staff. As an aside, when NIH PO1 grants moved from site visits to reverse site visits (that is, the applicants video-conferenced into the review team rather than the reviewers visiting the applicant institution for the evaluation), scores, enthusiasm and success rates dropped. Reduced investment = reduced enthusiasm.

Each of these positive points is suppressed by on-line peer review where the incentives to be attentive and the penalties for being disengaged are significantly reduced. The reduced investment for on-line review occurs throughout the process, which is, itself, then devalued by the reviewers themselves, as well as the applicants. Of course this doesn’t reduce the application pressure and so there are no outward signs of a problem. But peer review is qualitative: its entire raison d’etre is to afford scientific evaluation of the potential of a proposal and to weigh that against others. If value is extracted from the process, so will its merit.

So what are the solutions? Perhaps a granting agency will perform a thorough head-to-head comparison of quality. Until then, perhaps virtual and in-person reviewing could be alternated? I’ve a sense this would, in of itself, yield useful information while enjoying partial fiscal and convenience benefits. It seems a small price to pay as opposed to a real possibility of system-wide degradation of peer review quality and function. There is false economy in reducing investment in the peer review process. An average CIHR operating grant costs about $1 million but, more importantly, there is real opportunity cost if the best ideas are not supported. We are scientists, and surely we should not let processes that gate-keep what science is actually performed dodge scientific evaluation themselves?

Footnote 1. It could be argued that not all science needs, and may even be stifled by, peer review. Fred Sanger, double Nobel laureate, published few papers and was largely supported by an institute grant from the Laboratory for Molecular Biology. Peer review tends to reject ideas that don’t conform to dogma, yet those dogma-challenging ideas are how science is tested.

Footnote 2. The issues were not a reflection of use of ranking vs rating as other work has shown reviewers ranking can be more accurate than rating: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0292306