Peer review is an important part of scientific publishing, but has been criticised for being ineffective, slow and expensive.^1–3 Reviewers are usually unpaid and their efforts not formally acknowledged, but journal editors depend on them to inform publication decisions and to improve the quality of manuscripts. Editors of some journals have frequent difficulty finding appropriate reviewers who are able to complete timely reviews. This can result in publication delay.

Only a few empirical studies to date have attempted to find ways of improving quality of peer reviews.^4–6 Tested interventions include the provision of written feedback,⁴ attendance at training workshops^5,6 and the use of self-taught training materials,⁶ but none resulted in improvements in review quality. Other studies have evaluated methods of improving the timeliness of reviews.^7,8 Pitkin and Burmeister⁷ conducted a trial and found that requesting permission from reviewers before sending manuscripts for review led to a higher number of refusals than when manuscripts were sent before seeking permission. However, those who agreed to review after a request completed their reviews more quickly than those who had not been initially asked. In another study they found that contacting late reviewers resulted in a review within 7 days in about two thirds of cases, whether contact was by telephone, fax or email.⁸

We conducted a survey of reviewers for five biomedical journals to determine the reasons why they decline to review, and their opinions on reviewer incentives.

METHODS

Questionnaire development

The questionnaire content was developed from two sources.

Interviews with reviewers

We conducted 28 semistructured telephone interviews with reviewers from four biomedical journals owned by the BMJ Publishing Group to determine the range of reasons why reviewers decline requests to review and what rewards might encourage reviewers. We used a random sample of reviewers for the journals Archives of Disease in Childhood, BMJ, British Journal of Ophthalmology and Sexually Transmitted Infections, who had declined a request to review more than once during the previous 12 months. A series of open-ended questions was asked about the most recent occasion respondents accepted or declined to review for the journal during the study period. The questions covered factors important in accepting and declining to review, and opinions and preferences for reviewer incentives. Responses were recorded by the interviewer at the time of interview and later tabulated.

Reviewer feedback

The BMJ Publishing Group’s online reviewing system asks reviewers to state a reason each time they decline a request to review. On the basis of recurrent responses, we developed a set of key reasons.

Questionnaire construction

Items from both sources were grouped into four broad questionnaire themes: reasons why reviewers decline to review, reasons why reviewers accept to review, opinions on financial incentives and opinions on non-financial incentives. All items were constructed as 5-point Likert scales, with low scores indicating low importance or low agreement. The questionnaire was piloted and refined before use in the survey.

Survey

After receiving approval for the study from the BMJ Ethics Committee, we generated a list of all invited reviewers for the journals Archives of Disease in Childhood, BMJ, Emergency Medicine Journal, Gut and Journal of Epidemiology & Community Health for a 9-month period. We divided reviewers for each of the journals into strata on the basis of the number of times they had declined or accepted a request to review during this study period. We then randomly selected 200 reviewers from across the strata from the samples of each journal using the random number generator in MS Excel, giving a total sample of 1000. Invitations to take part in the survey were sent by email, including a link to access the online survey. Reminders were sent by email to non-responders at weeks 2 and 5 after the initial mail.

RESULTS

Response rates

We received 110 automated responses from invalid email addresses. We received a response from 606 of the 890 (68%) remaining valid email addresses. Fifty five respondents completed only the opening questions of the survey and were excluded from the analysis. We analysed results for the five journals individually (Archives of Disease in Childhood, n = 115; BMJ; n = 133; Emergency Medicine Journal, n = 101; Gut, n = 90; Journal of Epidemiology & Community Health, n = 112). However, results across journals were broadly similar, so we present only the overall findings from the five journals combined (n = 551).

Respondent characteristics

Of the 551 respondents, 30% (n = 386) of the reviewers were women, 71% (n = 389) were aged <55 years, 66% (n = 362) were academics and 88% (n = 486) were active writers of research articles.

Reasons for accepting to review

Data are presented as the mean (standard deviation (SD)) of responses on a 5-point Likert scale.

In all, 429 respondents reported they had accepted a request to review during the study period. Factors rated most highly in importance to the decision to accept a paper to review included contribution of the paper to the subject area (3.67 (0.86)), relevance of the topic to own work/interests (3.46 (0.99)) and opportunity to learn something new from the paper (3.41 (0.96); table 1). A sense of professional duty and the reputation of the journal were also considered important factors. Factors rated as least important were monetary payment (BMJ reviewers only, 1.64 (0.76)), the reputation of the authors of the paper (1.78 (0.95)) and academic reward such as career enhancement (1.81 (0.96)).

Reasons for declining to review

During the study period, 258 reviewers reported that they had declined a request to review. The most highly rated factor important in the decision to decline to review was conflict with other workload (4.06 (1.31), table 2). Having too many reviews for other journals (2.54 (1.45)) and a tight deadline for completing the review (2.70 (1.32)) were also important in the decision to decline. Factors rated as least important were having previously reviewed many papers on the same subject (1.26 (0.59)), length of the manuscript (1.29 (0.63)), comments not taken into account in the past reviewing experience (1.34 (0.81)), lack of formal recognition of reviewer contribution (1.38 (0.77)) and having to use the online review system (1.39 (0.92)).

Opinions on financial incentives

The survey yielded more agreement than disagreement that financial incentives will not be effective when time constraints are prohibitive (3.59 (1.01)) and that small financial incentives would not encourage reviewers to accept reviews (3.53 (0.95)), but many reviewers agreed that financial incentives encourage reviewers to accept requests to review (3.39 (0.86), table 3). There was little agreement that financial incentives could improve the quality of reviews (2.42 (0.98)), but responses to the item “financial incentives could compromise the quality of review” were mixed (3.08 (1.00)).

Opinions on non-financial incentives

We found highest agreement that the following incentives would encourage reviewers to accept requests to review: free access or subscription to journal content (3.72 (1.04)); annual acknowledgement on the journal’s website (3.64 (0.90)); more feedback about the outcome of the submission (3.62 (0.88)) and quality of the review (3.60 (0.89)); and appointment of reviewers to the journal’s editorial board (3.57 (0.99), table 4). There was less agreement that publication of the review with the article (2.62 (1.09)), the option of adding the reviewer’s name at the end of the published article (2.77 (1.10)) and gifts would act as incentives for reviewers.

DISCUSSION

Reviewers were more likely to accept a request to review a manuscript when the paper was relevant to their area of interest. Lack of time was the principal factor in the decision to decline. Participants regarded reviewing as an opportunity to learn something new in their area of interest and as part of their professional duty. They were not of the opinion that small financial incentives would be effective, but were favourable to the idea of some non-financial incentives. The most motivating incentives included free access or subscriptions to the journal, annual acknowledgement of reviewers on the journal’s website, feedback about the outcome of the manuscript submission and the quality of their reviews, and the appointment of the best reviewers to the journal’s editorial board.

Study implications

Peer review is an important and accepted part of scientific publishing and has become an indicator of perceived journal quality. Many reviewers regard journal peer review as part of their professional responsibility. However, academic institutions do little to train reviewers, and expect staff to review in addition to their formal workload. Although journal editors rely on reviewers to improve the quality of manuscripts and to inform decision making regarding the publication, reviewers are usually unpaid and their efforts not formally acknowledged by the journal. Interestingly, 80% of participants said that lack of formal recognition was not an important factor in their decision to decline. Some journals now offer reviewers continuing medical education or continuing professional development credits, but these are relevant only to those in medical posts.

Reviewers in our study indicated that they were not motivated by financial incentives, possibly because they saw reviewing as part of their academic role and were not usually paid for reviewing. Most journals cannot afford to pay reviewers consultancy rates for their time, but there are some inexpensive options that may act as acknowledgement or incentive for reviewers. For example, more journals could post annual acknowledgement lists of their reviewers on their websites, send letters of thanks to their most valued and regular reviewers, appoint their best reviewers to their editorial boards and give more feedback about the editorial outcome of the papers reviewed. However, although reviewers were favourable to the idea of feedback about their reviews, this would be resource intensive for journals.

Peer review has traditionally been a closed “black box” process. Some journals are considering a move to open peer review where the reviewers of published manuscripts are acknowledged in the publication and have the option of their review being made openly available. Reviewers in this study were divided over the idea of a published acknowledgment of the reviewer’s contribution and less keen on the publication of the review in the journal. Further research is necessary to gauge reviewers’ perceptions and concerns before switching to an open-review process.

With an increasing number of submissions to journals and demand on reviewers’ time, it is becoming increasingly important to solicit reviewers appropriately. Journals should remind reviewers to update their personal profiles, detailing their areas of expertise to ensure that they receive appropriate review requests and are not sent more than the agreed quota.

Study limitations

Our study was limited to a sample of reviewers with internet and email access as our survey was administered online. It is possible that reviewers who use electronic review systems have views different from those who use traditional paper-based reviewing. However, all journals published by the BMJ Publishing Group now require reviewers to use the journals’ online reviewing system, so respondents were representative of reviewers for these journals. Many other journals now use electronic tracking systems. It is also possible that reviewers working in specialties not covered by the journals in this study may have additional considerations when choosing whether or not to review for a journal. We deliberately selected journals with international reviewers and from different specialties, as well as a general medical journal, to increase the generalisability of findings across biomedical reviewers. Unfortunately personal characteristics of reviewers such as age, sex and reviewing experience are not recorded routinely on the manuscript tracking databases, so we were unable to compare responders with non-responders in terms of these factors.

Conclusion

Reviewing should be formally recognised by academic institutions, and journals should formally, and perhaps publicly, acknowledge the contribution of their reviewers.