ABSTRACT
Background Reviewers rarely comment on the same aspects of a manuscript, making it difficult to properly assess manuscripts’ quality and the quality of the peer review process. It was the goal of this pilot study to evaluate structured peer review implementation by: 1) exploring if and how reviewers answered structured peer review questions, 2) analysing reviewer agreement, 3) comparing that agreement to agreement before implementation of structured peer review, and 4) further enhancing the piloted set of structured peer review questions.
Methods Structured peer review consisting of 9 questions was piloted in August 2022 in 220 Elsevier journals. We randomly selected 10% of these journals across all fields and IF quartiles and included manuscripts that in the first 2 months of the pilot received 2 reviewer reports, leaving us with 107 manuscripts belonging to 23 journals. Eight questions had open ended fields, while the ninth question (on language editing) had only a yes/no option. Reviews could also leave Comments-to-Author and Comments-to-Editor. Answers were qualitatively analysed by two raters independently.
Results Almost all reviewers (n=196, 92%) filled out the answers to all questions even though these questions were not mandatory in the system. The longest answer (Md 27 words, IQR 11 to 68) was for reporting methods with sufficient details for replicability or reproducibility. Reviewers had highest (partial) agreement (of 72%) for assessing the flow and structure of the manuscript, and lowest (of 53%) for assessing if interpretation of results are supported by data, and for assessing if statistical analyses were appropriate and reported in sufficient detail (also 52%). Two thirds of reviewers (n=145, 68%) filled out the Comments-to-Author section, of which 105 (49%) resembled traditional peer review reports. Such reports contained a Md of 4 (IQR 3 to 5) topics covered by the structured questions. Absolute agreement regarding final recommendations (exact match of recommendation choice) was 41%, which was higher than what those journals had in the period of 2019 to 2021 (31% agreement, P=0.0275).
Conclusions Our preliminary results indicate that reviewers adapted to the new format of review successfully, and answered more topics than they covered in their traditional reports. Individual question analysis indicated highest disagreement regarding interpretation of results and conducting and reporting of statistical analyses. While structured peer review did lead to improvement in reviewer final recommendation agreements, this was not a randomized trial, and further studies should be done to corroborate this. Further research is also needed to determine if structured peer review leads to greater knowledge transfer or better improvement of manuscripts.
Competing Interest Statement
Mario Malički is Co-Editor-in-Chief of Research Integrity and Peer Review journal. BM is an employee of Elsevier, the publisher of the journals studied in this paper and the owner of the submission system used for piloting the structured peer review, and for collecting the data and reviewer responses that were analysed in this study. Elsevier is also the owner of Scopus database which was used to select journals from different impact factor quartiles and subject areas.
Footnotes
Code Availability: No code was used in this study. Statistical analysis outputs are available at Stanford Digital Repository.
Data Availability: As categories for outcome classification were developed throughout the data extraction process, the data file was constantly being updated. The final, cleaned dataset, is posted at Stanford Digital Repository. The original raw data, which contains proprietary Elsevier data, is available upon request from BM and will require a non-disclosure agreement.
Disclaimer: None.
Funding: None.
Ethics Approval: No ethics approval was needed as we analysed an anonymised dataset and all editors agreed to partake in the trial.
Presentations at meetings/conferences: We presented preliminary results as a poster at the Metascience 2023 Conference held on May 9-10, 2023. Washington, DC, USA. MM received a travel reward for postdocs that covered his registration fee and 300$ to help with travel and accommodation costs. We also presented preliminary results, as an oral presentation at the PubMet Conference, September 13-15, Zadar, Croatia.
Preprint availability: This is the first preprint version of the study.
Protocol and analysis plan availability: This study and its analysis plan were not pre-registered. Reporting: This study was reported following the Strengthening of the Reporting of Observational Studies in Epidemiology (STROBE) guidelines and Standards for reporting qualitative research (SRQR).
Statement of Interests: Mario Malički is Co-Editor-in-Chief of Research Integrity and Peer Review journal. BM is an employee of Elsevier, the publisher of the journals studied in this paper and the owner of the submission system used for piloting the structured peer review, and for collecting the data and reviewer responses that were analysed in this study. Elsevier is also the owner of Scopus database which was used to select journals from different impact factor quartiles and subject areas.
