Abstract
In efforts to increase efficiency and convenience and reduce administrative cost, some granting agencies have been exploring the use of alternate review formats, particularly virtual panels utilizing teleconference-based (Tcon) or Web based (Wb) technologies. However, few studies have compared these formats to standard face-to-face (FTF) reviews; those that have compared formats have observed subtle differences in scoring patterns and discussion time, as well as perceptions of a decrease in discussion quality in virtual panels. Here we present data from a survey of reviewers focused on their perceptions of the facilitation and effectiveness of panel discussion from their last peer review experience in virtual (Vcon/Tcon/Wb) or FTF panel settings. Reviewers indicated that, in terms of participation, clarifying differing opinions, informing unassigned reviewers and chair leadership, the facilitation of panel discussion was viewed similarly for FTF versus Vcon/Tcon reviewers. However, significant differences were found for many of these parameters between FTF and Wb reviews, which may suggest better panel communication (and thus more effective discussion) in FTF panels. Importantly, these differences disappear for respondents with high frequency of review participation (likely study section members), suggesting perhaps that long standing teams may be more resilient in virtual communication settings. Surprisingly, no significant differences were found between any of the reviewers’ experiences in virtual or FTF settings in terms of the discussion affecting the outcome, in choosing the best science, or even whether the discussions were fair and balanced. However, significant differences were reported between Wb and FTF reviewers in terms of their perceptions of how well their expertise (as well as the expertise of their fellow reviewers) was utilized on the panel, which may suggest that the level of communication provided in FTF panels allows for better integration of expertise across panel members when evaluating research proposals as a team. Again, these differences across review setting disappeared for respondents that reviewed more often. Overall, despite clear preferences by reviewers for FTF panels, the lack of differences between FTF and Vcon/Tcon panel facilitation or discussion quality potentially supports the use of this review format by granting agencies, although subtle differences may exist that were not reported by reviewers in this survey. These results also provide some evidence of the perceived limitations in discussion quality in Wb panels, at least in non-recurring panels.
Introduction
The US National Institutes of Health (NIH), like many major research funders, utilizes a “long standing and time tested system of peer review to identify the most promising biomedical research (NIH 2018).” However, peer review is implemented in a variety of ways and meeting formats by different funding agencies and institutes (Liaw 2017). Even across the NIH, to improve the efficiency, cost-effectiveness, and convenience of the process, panel meetings meet not only face-to-face (FTF), but sometimes via videoteleconference (Vcon) or through a Web based review (NIAID 2018). However, these alternate review formats have not completely replaced in-person meetings, in part due to reviewer preferences toward FTF formats (Gallo 2018). In fact, when the Canadian Institute of Health Research (CIHR) almost completely replaced all face-to-face review meetings with virtual ones, and there was a significant backlash from the scientific community, which was of the opinion that the quality of the decision making on virtual panels was much lower (Webster 2015). Eventually, ignoring the recommendations from a report from an international working group suggesting that the “asserted benefits of face-to-face peer review are overstated” (Gluckman 2017), CIHR relented and abandoned its reforms, focusing again on in-person review meetings (Webster 2017).
What is striking about these policy shifts is the scant evidence supporting the use of one review format over another. The literature surrounding grant peer review as a whole is very limited, and while some studies in the literature have examined review panel discussion and its effects on scoring (Obrecht 2007, Martin, 2010, Fogelham 2012, Fleurence 2014, Forsythe 2018), only four have contrasted traditional and alternate review formats (Gallo 2013, Carpenter 2015, Pier 2015, Vo and Trocki 2015). While Gallo et al. (2013) has found no significant differences between face-to-face (FTF) or teleconference (Tcon) panels in terms of the average, breadth or levels of contentiousness in the final scores, both Pier et al (2015) and Carpenter et al. (2015) noted that the effect of discussion on scoring (shifts in scoring by assigned reviewers after discussion) was slightly but significantly muted in Tcon panels as compared to FTF panels (Vcon panels in the case of Pier et al). Consistent with previous findings, these analyses found the magnitude of these scoring shifts after discussion were small and only affected the funding status of a small portion of grants. In addition, these four studies found the average discussion time was reduced for Tcon/Vcon panels, although no correlation between discussion time and the magnitude of shifting scores post-discussion were found in any of these studies. Finally, a 2015 NIH survey of reviewers found that the quality of discussions for text-based review was not rated as highly as that of FTF or even Tcon/Vcon reviews (NIH 2015). In addition, one study found 43% of reviewers found virtual review panels to have minimal interaction among reviewers (Vo and Trocki 2015).
These results point to a slight reduction in reviewer engagement in virtual panels, which is consistent with the literature on distributed teams (Rogelberg 2002). Lowered engagement and reduced levels of trust among virtual team members is well documented, more so in text based communication (Driskoll 2003, Zheng 2002, Cooke 2015, Bos 2002). In addition to the lack of visual cues, opportunities to generate intra-panel trust during panel breaks and meals are also missing in virtual settings (Blatner, 2009). It has also been suggested that virtual teams may have more difficulty developing transactive memory, including an understanding of the location of expertise within the panel (Kanawattanachai and Yoo, 2007), which in peer review may reduce productive participation from unassigned panel members. Finally, persuasive tasks, which are crucial to review discussions, have been shown to be particularly affected by communication setting (Driskoll 2003).
It is unclear how precisely the reduced engagement seen in virtual panels manifests itself in terms of the review team processes. For a review panel discussion to work effectively, there must be a sense of inclusion across all panel members (such that reviewers feel enabled to lend their expertise to the discussion). Is the decrease in discussion times observed in virtual panels due to lowered engagement of unassigned reviewers, assigned reviewers, or both? Also, arguments about the quality of research proposals must be clearly communicated to be persuasive, yet it is unclear if virtual communication hinders the clarity of discussions or the persuasiveness of the arguments (as seems may be the case given the reduced levels of scores shifting post-discussion). Moreover, it is also unclear if team leadership is hindered in a virtual review format, thereby limiting the chair’s ability to facilitate the discussions. Ultimately and most importantly, are the discussions of similar quality across review formats and do they equally promote the best science? These types of questions are not easily answered through the analysis of scores. Also, Pier et al. (2015) suggests that despite the relative similarities in scoring, reviewers perceive currently unmeasured benefits of FTF meetings including “the camaraderie and networking that occurs in person, the thoroughness of discussion, the ease of speaking up or having one’s voice heard, the fact that it is more difficult to multi-task or become distracted, reading panelists’ facial expressions, and perceived cohesiveness of the panel.”
Recently, the American Institute of Biological Sciences (AIBS) developed a survey to address reviewer perceptions of their most recent panel meeting experience and distributed it to biomedical scientists. Two publications (Gallo 2018a,b) have resulted from analysis of the survey responses, however neither addressed discussions quality, despite having included a section in the survey on discussion facilitation and its impact on review outcomes. To examine some of these questions posed above, the AIBS analyzed feedback from these scientists about the quality and facilitation of most recent panel discussions and asked respondents to indicate whether their meeting format was FTF, Vcon/Tcon or Wb in the hope to shed some light on the effect of the panel meeting setting on review effectiveness and quality. Further, an understanding of reviewer perceptions of panel effectiveness may inform the future implementation of different review formats, which for now may be largely driven by cost-savings incentives.
Survey Methodology
As mentioned, the general survey methodology has been described in two other manuscripts (Gallo 2018a, Gallo 2018b). The original survey contained 60 questions and was divided into 5 subsections (the full survey is available in the Appendix); however, only 3 sections will be analyzed in this manuscript to address the issue of discussion quality: 1. Grant Submission and Peer Review Experience, 2. Reviewer Attitudes toward Grant Review and 3. Peer review panel meeting proceedings. The questions regarding discussions quality included here were not analyzed in the previous publications, although other aspects included here, like review frequency and reviewer preference were looked at previously. It should also be noted that the survey was reviewed by the Washington State University Office of Research Assurances (Assurance# FWA00002946) and granted an exemption from IRB review (IRB#15268; 45 CFR 46.101(b)(2)).
The questions examined had either Yes/No or Likert interval rating scale (1 to 5, where 1 is most positive and 5 is most negative) response choices; for example, “on a scale of 1-5 (1 most definitely, 5 not at all), did the grant application discussions promote the best science?”). However, respondents were also given the choice to select “no answer/prefer not to answer.” At the end of each section, respondents could reply in free form text to clarify answers. A full copy of the peer review survey is available in the Appendix.
As mentioned in previous publications, the survey was sent out in September of 2016 to 13,091 individual scientists from AIBS’s database through the use of Limesurvey, which de-identified the responses from respondents. AIBS’s proprietary database has been developed over several years to help AIBS recruit potential reviewers for evaluation of biomedical research applications for a variety for funding agencies, research institutes and non-profit research funders. Most of these reviews are non-recurring and scientists are recruited based on matching expertise to the topic areas of the applications. All individuals participating in this survey were either reviewers for AIBS (36%) or had submitted an application as a PI which was reviewed by AIBS (71%) or both (12%). Respondents were asked to answer questions based on either the most recent peer review or reviews that occurred in the last 3 years (depending on the question); these reviews did not have to be AIBS reviews (it is likely that the majority of reviews reported here were not for AIBS).
The survey was open for two months; responses were then exported and analyzed through basic statistical software. For this analysis, participant responses were included only if they were fully submitted and included an answer for question 2e, 2.f. and 2.g., which focused on whether they had participated in a peer review panel in the last three years, and if so how often. Thus, all questions included in this analysis were focused on reviewer experiences. Reviewers were asked questions related to the qualities of panel discussion and expertise. The data were separated out by reviewers’ recent review format (FTF, Vcon/Tcon and Wb) and answers to questions on reviewer experience were compared.
Results
Response Rate and Demographics
Of the 13,091 individuals contacted for this survey, 1231 responded, giving a 9.4% response rate. However, only 874 of these completed questions 2e, 2f, and 2g, 671 (77%) of whom indicated they had recently reviewed on a panel in the last three years. These 671 reviewer respondents formed the core group upon which the current analysis is based. Demographics were analyzed in detail in previous publications. Respondents were 66% male, 80% PhD and 69% in a late career stage (e.g. Tenured Full and Emeritus Professorship) with 75% being age 50 or older. Also, the majority of respondents (76%) were Caucasian and worked in Academia (81%). These respondents participated in an average of 4.0 ± 0.08 panel meetings over the last three years and, similar to our previous analysis (Gallo 2018b), the distribution of their participation was bimodal, with 155 (23%) respondents participating in 7 or more reviews in a three-year time frame (Rev7 respondents), and the other 516 (77%) reviewer respondents participating in 6 or fewer reviews in this time (non-Rev7).
Review Setting
Of the reviewer respondents, 49% (N=331) recently participated in a FTF meeting, 26% (N=172) took part in a Vcon/Tcon meeting, 22% (N=148) took part in a Wb meeting while 3% marked “other” for review setting. When reviewers were asked which review setting they preferred, 80% (N=516) indicated FTF, while only 11% (N=73) indicated Vcon/Tcon and 9% (N=57) indicated Wb (26 indicated no answer). For the clear majority of reviewers that preferred in-person meetings, our previous publication reports that reviewers selected communication as the most influential reason for their preference in review format; for reviewers who preferred virtual meetings, logistical convenience was the most important determinant (Gallo 2018b). It should be noted that reviewer preference for review format was related to reviewer experience; 61% of reviewers who preferred FTF were involved in a recent FTF panel, compared to only 17% of reviewers who preferred virtual formats. The frequency of review participation was also related to review format; 67% (104) of Rev7 reviewers recently participated in a FTF panel compared to 44% (227) of non-Rev7 reviewer respondents. As was noted in a previous publication, Rev 7 respondents are likely associated with membership in study sections, which meet several times a year (more often by FTF) and have memberships which may last a couple of years (Gallo 2018b).
Panel Expertise
The majority of reviewers felt their own expertise as well as that of the other panel members was either definitely or most definitely well utilized (82% and 74% for their own expertise versus panel expertise, respectively, for all reviewers). However, reviewers felt more positive about the utilization of their personal expertise as compared to that of other panel members (t[1285]=3.93, p<0.001). As listed in Table 1, FTF reviewers most strongly felt that their scientific expertise was necessary and appropriately used in the review process, more so than reviewers in Vcon/Tcon or Wb settings. In person reviewers also felt, significantly more so than Wb reviewers, that the expertise of the other panel members was necessary and appropriately used in the review process.
We also analyzed just the Rev7 respondent population to see if long-term panel membership may be affect perceptions of expertise utilization. We observed that Rev7 reviewers perceive no difference in the utilization of their own expertise between FTF and IA panels or FTF and Tcon panels (Table 2). In fact, Rev7 respondents generally find their own expertise better utilized than non-Rev7 reviewers (t[627]=3.72; p<0.001). Similarly, Rev7 reviewers perceive the use of other panel member’s expertise equivalently in both FTF and IA panels as well as in FTF and Tcon panels (Table 2). Again, Rev7 respondents find panel expertise better utilized than non-Rev7 reviewers (t[620]=2.05; p<0.05).
Discussion Facilitation
As listed in Table 3, the vast majority (89%-94%) of reviewers felt the panel discussions facilitated reviewer participation and this did not vary significantly across review settings (although the difference between FTF and Wb did approach significance). Similarly, 70% of all reviewers felt discussions were most useful or very useful in clarifying opinions. However, FTF reviewers did feel the discussions were more useful in clarifying differing reviewer opinions than Wb reviewers; no differences were observed between FTF and Vcon/Tcon reviewers (Table 3). While most reviewers (69%-82%) agreed that the format and duration of the grant application discussions was sufficient to allow the non-assigned reviewers to cast well informed merit scores, a significantly higher proportion of FTF reviewers as compared to Wb reviewers felt this way (Table 3). Again, no differences were found between FTF and Vcon/Tcon reviewers. In terms of the usefulness of the chair in facilitating the application discussions, 68% of all reviewers reported that the chair’s involvement was either extremely useful or very useful. Again, FTF reviewers were more likely than Wb reviewers (but not Vcon/Tcon reviewers) to feel that the chair was useful in facilitating discussions (Table 3).
Interestingly, respondent preference for review format did not influence perceptions of discussion facilitation. For example, of all respondents that recently experienced a virtual meeting (Vcon/Tcon/Wb), 91% (171) of those who preferred FTF meetings and 87% (84) of those who preferred virtual (Vcon/Tcon/Wb) meetings felt the discussions facilitated participation, displaying no significant differences (X2[1]=1.6; p>0.05). Similarly, of respondents that recently experienced a Vcon/Tcon/Wb meeting, 73% (131) of those who preferred FTF meetings and 78% (72) of those who preferred Vcon/Tcon/Wb meetings felt the format and duration of the discussions was sufficient to allow the non-assigned reviewers to cast well informed merit scores (X2[1]=0.97; p>0.05).
Importantly, no differences in the perception of discussion facilitation were found among Rev7 reviewers in terms of the discussion clarifying different opinions, allowing unassigned reviewers to make informed decisions or the usefulness of the chair in facilitating discussion (Table 4). However, Rev7 reviewers did report that FTF reviews better facilitated participation in discussion than either Vcon/Tcon or Wb panels (Table 4). It should be noted that there were no significant differences in perception between Rev7 and non-Rev7 reviewers in terms of the discussion facilitating participation (X2[1]=0.69, p>0.05), allowing unassigned reviewers to cast informed scores (X2[1]=1.60, p>0.05), and clarifying differing opinions (t[640]=0.29; p>0.05), as well as the usefulness of the chair (t[637]=0.18; p>0.05).
Discussions and Outcome
A total of 71% of reviewers agreed that panel discussion was extremely effective or very effective in influencing the outcome of the grant, although no differences were found across review setting (Table 5). Similarly, 60% of reviewers definitely or most definitely agreed that the grant application discussions promoted the best science; again no differences were found across review setting (Table 5). Interestingly, reviewers were significantly more positive that the discussions were affecting the outcome than they were facilitating the selection of the best science (t[1276]=3.53, p<0.01). Finally, while the overwhelming majority (87%-88%) of reviewers felt the discussions were fair and balanced, review format did not affect the perceived fairness of the discussion (Table 5).
Similarly, Rev7 reviewers find no difference between FTF and IA discussion or FTF and Tcon discussion in terms of affecting the outcome, promoting the best science and whether discussions were fair and balanced (Table 6). Moreover, there were no significant differences between Rev7 and non-Rev7 reviewers (Table 6) in terms of affecting the outcome (t[636]=0.57; p>0.05), promoting the best science (t[638]=0.44; p>0.05) or for whether the discussions were fair and balanced (X2[1]=0.54, p>0.05).
Discussion
Our results indicate a clear preference for FTF panels by respondents, and our previous publication suggests this is largely due to the perceived quality of communication in FTF panels; those who prefer virtual panels suggest logistical convenience as an important motivation (Gallo 2018). Thus, it is unsurprising that reviewer preference and reviewer experience were found to be related, where respondents who preferred FTF panels were much more likely to have recently participated in a FTF panel as compared to those who prefer virtual panels. It is interesting that these preferences did not seem to have a strong bearing on how reviewers felt about the quality of panel discussion, suggesting that the responses recorded here are more linked to their actual experiences than to any pre-conceived notions of peer review.
We also observed that reviewers generally felt their own expertise as well as that of other panel members was well utilized, although they felt more positive about their own expertise as compared to that of other panel members (Table 1). Others have reported that individual openness to the diversity of team expertise affects team performance (Homan 2008); thus, it may be that the differences found here are related to different degrees of openness amongst reviewers, where perhaps a small proportion of respondents are truly open to the multiplicity of panel expertise. Interestingly, Rev7 respondents (presumably study section members) generally find their own expertise and the expertise of other panels members better utilized than non-Rev7 reviewers (Table 2). This is likely the result of long-standing team members having better knowledge how their (and others’) expertise fit into the decision making process as compared to ad-hoc reviewers and teams.
Overall, significant differences were found across review setting for both of these measures of expertise (Table 1); not only between FTF and Vcon/Tcon and FTF and Wb reviewers in terms of utilization of their individual expertise but also panel expertise between FTF and Wb reviewers. These differences with review settings disappear for Rev7 reviewers (Table 2). Again, it may be this relates to the panel effectiveness of deep knowledge integration of the collective team expertise, which may vary considerably depending on review setting and length of time the team has been together (Brooke, 2015). Poor integration may lead to not only to poorly perceived utilization of group expertise, but also of how an individual’s expertise fits into the group. As mentioned, virtual teams may have difficulty in developing an understanding and trusty of where expertise is distributed across the panel, which may negatively influence perceptions of its effective use by the panel (Kanawattanachai and Yoo, 2007). Thus, while it likely can be assumed reviewers are recruited in a similar way for FTF and Wb and thus expertise is similarly matched to proposals, it may be that richer communication channels provided in FTF panels allow for better knowledge integration across team membership, which leads to a better appreciation of each other’s expertise (particularly for ad-hoc teams). It is likely this is compensated for in long standing teams by the strengthening of knowledge integration of team expertise over time. Previous results from a survey of NIH reviewers also found only small differences between FTF (89%) and Vcon (81%)/Tcon (82%) reviewers in terms of the proportion that regarded the adequacy of the panel expertise favorably (no test for significance); unfortunately they did not include Wb meetings in this measure (NIH 2015).
Wb reviewers were also more negative about the facilitation of review discussions as compared to FTF reviewers (Table 3). Wb reviewers were less likely than FTF reviewers to find the discussions useful in clarifying differing opinions or sufficient to allow un-assigned reviewers to make well informed judgements. They were also less likely to find the chair to be a good facilitator of those discussions. Thus, it seems reviewers who recently participated in a Wb meeting are less likely to find the team communication clear, inclusive, and well facilitated. Again, these differences were not found among Rev7 reviewers (Table 4). If these long-term teams have generated significant trust across its membership (as compared to ad-hoc panels), this may provide an advantage in enhancing communication and facilitating more useful discussions, even in Wb panels. However, results from the 2015 NIH survey indicated smaller proportions of reviewers who had favorable impressions of discussion quality with Vcon (70%)/Tcon (76%) and Wb (67%) reviewers compared to FTF (83%), although again no tests for significance were reported (NIH 2015). It is unclear why our Rev7 results differ from those from NIH, as it is likely they are the same population. One difference is that we asked 4 specific questions about aspects of discussion facilitation while NIH asked whether “discussions supported the ability of the panel to evaluate the applications being reviewed,” which is more general and may have wrapped many of these aspects together, potentially yielding a larger effect. In addition, the NIH survey was provided to reviewers during the panel meetings capturing the perceptions of reviewers right after the review, unlike our study which was distributed to all respondents at the same time and was not linked to their participation in a panel.
Small differences (which approached significance) were seen between Wb reviewers and FTF reviewers in terms of reviewer participation, and significant differences did exist among Rev7 reviewers between FTF and Vcon/Tcon panels as well as between FTF and Wb panels (Table 4). Again, this may be due to potentially longer team membership and likely higher level of engagement during discussion (compared to ad-hoc reviewers). Study section members may be more likely to notice if there are lowered levels of participation in the discussion in virtual review settings, as they have something to compare it to (presumably the same panel also meets FTF).
Thus, in several areas we have observed differences in how reviewers perceive the facilitation of discussion in FTF panels compared to Wb panels. Moreover, the 2015 NIH survey results suggest much lower reviewer comfort levels with potentially having their own applications reviewed via a Wb panel versus a Vcon/Tcon panel. Taken together, these results are supported by the team science literature which suggests virtual team members in text only communication situations have great difficulty in developing team trust, even when compared to Vcon/Tcon teams, and need richer forms of communication to participate in cooperative tasks (Bos, 2002, Cooke, 2015). Generally, in our results Rev7 reviewers seem to be resistant to the communication limitations in virtual settings, perhaps because they have higher trust developed across their teams due to longer term membership, although some of this data is in contrast to NIH results (NIH 2015). Further research should explore this further by directly comparing the perceptions of ad-hoc and long-term panel members surrounding the quality of review discussion.
Interestingly, we found no significant differences in reported discussion facilitation between FTF and Vcon/Tcon review formats (Table 3). While we and others have previously found subtle differences in scoring and the length of discussion times between Tcon and FTF settings (Carpenter et al., 2015, Pier et al., 2015), this doesn’t seem to affect reviewer perceptions of how the discussion was facilitated, although the differences found between FTF and Wb settings underscores the importance of at least audio-facilitated communication and discussion.
The impact of the review discussion on the outcomes or in promoting the best science did not seem to be affected by the review setting, including Wb settings (Table 5), and this was the case for Rev7 reviewers as well (Table 6). It is particularly surprising that the review setting does not affect the perceived impact of discussion on review outcomes, given the clear differences observed here between Wb and FTF reviewers regarding the facilitation of discussions as well as previous data suggesting that post-discussion shifts in score are reduced in Tcon panels compared to FTF panels (Carpenter et al., 2015). It may be that the some of the effects of review setting on proposal discussion are more subtle than can be detected by reviewers. It may also be that reviewers are overconfident in the effectiveness of panel discussion, potentially because they were directly involved in this discussion (Moore and Healey, 2007). Future studies should examine actual panel discussions and analyze for potential linguistic and stylistic differences in FTF and Vcon/Tcon panels that may help explain the previously reported scoring differences (Raclaw 2017, Pier 2019). It would also be interesting to gather perceptions from outside panel observers (like scientific review officers who manage panels for funding agencies) which may counter reviewer perceptions, as well as compare reviewer survey responses to actual scoring data, both of which may serve to assess for overconfidence.
Interestingly, reviewers are significantly more positive about the discussions affecting the outcome than they are about selecting the best science (t[1276]=3.53, p<0.01). This may be related to the natural rater variability in assessing research quality that is inherent in peer review (Cole 1981, Pier 2018) and which likely exists independent of communication setting. Importantly, the vast majority of reviewers did feel that these panel discussions were fair and balanced, and this was found irrespective of review setting, which at least alleviates some of the concern that certain review formats promote bias more than others. However, it should be said that implicit bias may be a very difficult thing for reviewers to detect in a panel discussion, yet it may still have an important impact on panel discussion and scoring. Future work should more rigorously evaluate the relationship between implicit reviewer biases, panel discussion and review format.
Overall, while our reviewer pool indicated a clear preference for FTF panels, perceptions of Vcon/Tcon discussion quality was similar to that of FTF discussion quality; they were viewed as equally clear, inclusive and impactful, independent of reviewer preference. The previous scoring differences reported aside, it seems our results help bolster the case for Vcon/Tcon panels. However, it is also clear that reviewers do not feel the same way about the discussion quality of Wb panels and given their low popularity, much more justification should be sought before routinely implementing this review format. Finally, in terms of review formats that most efficiently avoid bias and promote the best science, from our results, no format seems to be particularly advantageous. However, future studies of discussion quality across review formats will need to account for the great variability in reviewer personality and panel leadership. For instance, variability in discussion time may be a function of chair behavior (limit-setting versus allowing discussion). Also, are more persuasive reviewers hindered more by review format than less proactive reviewers? Some have reported the importance of score-calibration comments and even laughter in the effectiveness of panel discussion, although it is unclear if these are affected in any way by review format (Pier et al., 2019). And as discussion has traditionally affected the funding status of only a small proportion of proposals (Martin 2010, Fogelholm 2012, Carpenter 2015), these types of studies should be examined in parallel with those examining the decision making processes that occur at the individual reviewer level.
Appendix (Full Survey)
Survey is listed below.
[]What is your gender?
Please choose all that apply:
Male
Female
Prefer not to answer
[]What is your age?
Please choose only one of the following:
Under 30
30–39
40–49
50–59
60+
[]Please specify your race/ethnicity
Please choose all that apply:
American Indian or Alaska Native
Asian or Asian American
Black or African American
Hawaiian or Other Pacific Islander
Hispanic or Latino
Non-Hispanic White/Caucasian
Other
Prefer not to answer
[]What type of degree(s) do you have?
Please choose all that apply:
PhD or other research doctorate
MD
DDS
DVM or VMD
Other
Prefer not to answer
[]What type of an organization do you work for?
Please choose only one of the following:
Academia
Government
Industry
Other
[]What stage of career have you reached?
Please choose only one of the following:
Early career
Mid career
Late career/tenured
Emeritus
[]On average, how many hours do you work each week?
Please choose only one of the following:
40 h
40–50 h
50–60 h
60–70 h
70 + h
[]Please provide any comments that justify your responses under Section 1, Demographics.
Please write your answer here:
Section 2: Grant submission and peer review experience
[]Have you submitted a grant for peer review in the last 3 years?
Please choose only one of the following:
Yes
No
[]If you answered yes to submitting a grant for peer review in the past 3 years, how many grant applications have you submitted in that time frame?
Please choose only one of the following:
1
2
3
4
5
6
7 or more
[]Did you receive reviewer feedback on your last grant submission?
Please choose only one of the following:
Yes
No
[]Was your last application successful, i.e., were you funded?
Please choose only one of the following:
Yes
No
[]Have you served on a peer review panel in the last 3 years?
Please choose only one of the following:
Yes
No
[]If you answered yes to serving on a peer review panel in the last 3 years, how many peer review panels have you served on in that time frame?
Please choose only one of the following:
1
2
3
4
5
6
7 or more
[]If you answered yes to serving on a peer review panel in the past 3 years, please select the mode of your last peer review panel meeting.
Please choose only one of the following:
Face-to-face
Remote (video/teleconference)
Internet-assisted
Other
[]How many ad-hoc reviews (usually one or two grant applications reviewed telephonically that are being evaluated in a panel meeting setting) have you performed in the past 3 years?
Please choose only one of the following:
0
1
2
3
4
5
6
7 or more
[]Have you reviewed for a journal in the last 3 years?
Please choose only one of the following:
Yes
No
[]If you answered yes to reviewing for a journal in the past 3 years, how many submissions have you reviewed in that time frame?
Please choose only one of the following:
1
2
3
4
5
6
7 or more
[]What is a higher personal priority: grant review or journal review?
Please choose only one of the following:
Grant review
Journal review
Both are equal priority
Neither is a priority
[]Please elaborate on your responses under Section 2, Grant submission and peer review experience.
Please write your answer here:
Section 3: Investigator attitudes toward grant review
If you answered yes to receiving feedback on your last grant submission, please answer Section 3 of the questionnaire. If you answered no, please proceed to Section 4.
[]On a scale of 1–5 (1 most useful, 5 least useful), overall how useful was the reviewer feedback you received on your last grant submission?
Please choose only one of the following:
1
2
3
4
5
[]On a scale of 1–5 (1 most useful, 5 least useful), how useful was the reviewer feedback in improving your grantsmanship?
Please choose only one of the following:
1
2
3
4
5
[]If you were not funded, on a scale of 1–5 (1 most useful, 5 least useful), how useful was the reviewer feedback in improving your future submissions?
Please choose only one of the following:
1
2
3
4
5
[]On a scale of 1–5 (1 most useful, 5 least useful), how useful was the reviewer feedback in informing your future scientific endeavors in the proposed research area?
Please choose only one of the following:
1
2
3
4
5
[]Did you feel the reviewer feedback was well written, cohesive, and balanced?
Please choose only one of the following:
Yes
No
[]Did you feel the reviewer feedback was fair and unbiased?
Please choose only one of the following:
Yes
No
[]Overall, in what area(s) did the reviewer feedback primarily focus?
Please choose all that apply:
Potential impact of research
Hypothesis
Research methodology
Innovation potential
Preliminary data
Responsiveness to funding mechanism
Statistical issues
Qualifications of research team
Budget
Other
[]Did the reviewers comment on the riskiness of the research project?
Please choose only one of the following:
Yes
No
[]Based on the reviewer feedback you received, do you feel that the reviewers had the appropriate expertise to evaluate your grant application?
Please choose only one of the following:
Yes
No
[]Please elaborate on your responses under Section 3, Investigator Attitudes Towards Grant review.
Please write your answer here:
Section 4: Reviewer attitudes towards grant review
[]What are your reasons for accepting an invitation to serve on a peer review panel?
Please choose all that apply:
Desire to give back to the scientific community
Networking opportunities
Informing your own grantsmanship
Gaining exposure to new and innovative scientific areas
Enhancing your career/resume
Expectation from the funding agency
Honorarium
Other
[]Do you feel that serving as a reviewer on peer review panels has positively impacted your career?
Please choose only one of the following:
Yes
No
[]If you feel that serving as a peer reviewer has positively impacted your career, in what ways has serving as a reviewer influenced your career?
Please choose all that apply:
Bolstered your career
Improved your grantsmanship
Increased your exposure to new scientific ideas
Improved your networking/collaboration opportunities
Other
[]In general, which type of panel meeting format do you prefer?
Please choose only one of the following:
Face-to-face
Virtual [teleconference/videoconference]
Internet-assisted
[]On a scale of 1–5, (1 most influential, 5 least influential), please rate the following factors in influencing your selection of preferred panel meeting format:
Please write your answer(s) here:
Logistical convenience
Level of communication among panel members
Networking opportunities
Likelihood to participate on panel
[]In the last 3 years, how many times have you declined an invitation to serve on a peer review panel?
Please choose only one of the following:
1
2
3
4
5
6
7 or more
[]What were your reasons for declining an invitation to serve on a peer review panel?
Please choose all that apply:
Limited free time Poor expertise match
Personal reasons (holiday, sickness, travel)
Review timeline too compressed
Conflict of interest
Issue with funding agency
Other
[]What is the maximum number of peer review panels/committees you prefer to serve on per year?
Please choose only one of the following:
1
2
3
More than 3
[]What is the maximum number of days you prefer to attend a peer review panel meeting? Please choose only one of the following:
1
2
3
More than 3
[]What is the maximum number of R01-type grant applications you prefer to be assigned for a peer review panel meeting?
Please choose only one of the following:
1–2
3–4
5–6
7
More than 7
[]What was the actual number of days of your last peer review panel meeting?
Please choose only one of the following:
1
2
3
More than 3
[]What was the actual number of R01-type grant applications you were assigned to review at your last peer review panel meeting?
Please choose only one of the following:
1–2
3–4
5–6
7–8
More than 8
[]On average, how many hours did you spend reviewing each grant application before the panel meeting?
Please choose only one of the following:
1–2
2–3
3–4
4–5
5–6
7 or more
[]Please elaborate on your responses under Sect. 4, Reviewer attitudes towards grant review. Please write your answer here:
Section 5: Peer review panel meeting proceedings
[]Please answer the following questions in relation to your last peer review meeting. On a scale of 1–5 (1 most definitely, 5 not at all), was your scientific expertise necessary and appropriately used in the review process?
Please choose only one of the following:
1
2
3
4
5
[]On a scale of 1–5 (1 most definitely, 5 not at all), from your perspective was the expertise of the other panel members necessary and appropriately used in the review process?
Please choose only one of the following:
1
2
3
4
5
[]Did the grant application discussions facilitate reviewer participation?
Please choose only one of the following:
Yes
No
[]Were the grant application discussions fair and balanced?
Please choose only one of the following:
Yes
No
[]On a scale of 1–5 (1 most useful, 5 least useful), how useful were the grant application discussions in clarifying differing reviewer opinions?
Please choose only one of the following:
1
2
3
4
5
[]On a scale of 1–5 (1 extremely effective, 5 no effect), did the grant application discussions affect the outcome?
Please choose only one of the following:
1
2
3
4
5
[]On a scale of 1–5 (1 most appropriate, 5 least appropriate), were the evaluation criteria appropriate to judge the best science and move the field forward?
Please choose only one of the following:
1
2
3
4
5
[]On a scale of 1–5 (1 extremely important, 5 of no importance), how important is the PI’s track record to assessing an investigator initiated (R01)-type application?
Please choose only one of the following:
1
2
3
4
5
[]In general, does a PI’s track record temper your assessment of any detected methodological weaknesses?
Please choose only one of the following:
Yes
No
[]On a scale of 1–5 (1 most definitely, 5 not at all), did the grant application discussions promote the best science?
Please choose only one of the following:
1
2
3
4
5
[]Was innovation factored into selecting the best science?
Please choose only one of the following:
Yes
No
[]Did you view innovation as an essential component of scientific excellence when evaluating the grant applications?
Please choose only one of the following:
Yes
No
[]Did the risks associated with innovative research impact the scores you assigned to the grant applications?
Please choose only one of the following:
Yes
No
[]On a scale of 1–5 (1 completely, 5 not at all), how much did the seniority of your fellow panel members influence your evaluations during the panel deliberations?
Please choose only one of the following:
1
2
3
4
5
[]Was the format and duration of the grant application discussions sufficient to allow the non-assigned reviewers to cast well informed merit scores?
Please choose only one of the following:
Yes
No
[]On a scale of 1–5 (1 extremely useful, 5 not useful at all), how useful was the Chair in facilitating the application discussions?
Please choose only one of the following:
1
2
3
4
5
[]Please elaborate on your responses under Section 5, Peer review panel meeting proceedings.
Please write your answer here:
Thank you for taking the time to fill out the survey. Have a wonderful day!
Submit your survey.
Thank you for completing this survey.