Hostname: page-component-8448b6f56d-tj2md Total loading time: 0 Render date: 2024-04-23T12:24:14.581Z Has data issue: false hasContentIssue false
  • English
  • Français

Commensuration Bias in Peer Review

Published online by Cambridge University Press:  01 January 2022

Carole J. Lee
Get access Rights & Permissions [Opens in a new window]

Abstract

To arrive at their final evaluation of a manuscript or grant proposal, reviewers must convert a submission’s strengths and weaknesses for heterogeneous peer review criteria into a single metric of quality or merit. I identify this process of commensuration as the locus for a new kind of peer review bias. Commensuration bias illuminates how the systematic prioritization of some peer review criteria over others permits and facilitates problematic patterns of publication and funding in science. Commensuration bias also foregrounds a range of structural strategies for realigning peer review practices and institutions with the aims of science.

Type
Social Sciences and Policy
Information
Philosophy of Science , Volume 82 , Issue 5 , December 2015 , pp. 1272 - 1283
Copyright
Copyright © The Philosophy of Science Association

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

†

I am grateful for the Career Enhancement Fellowship (funded by the Mellon Foundation and administered by the Woodrow Wilson Foundation) and Royalty Research Fund Scholars grant #A79071 (from the University of Washington), which afforded research time to complete this article. For helpful references and/or comments, I would like to thank Catherine Elgin, Elena Erosheva, Chris Haufe, John Manchak, Miriam Solomon, Kyle Stanford, and audience members at my 2013 American Association for the Advancement of Science presentation of this material.

References

Bornstein, Robert F. 1990. “Publication Politics, Experimenter Bias and the Replication Process in Social Science Research.” Journal of Social Behavior and Personality 5:71–81.Google Scholar
Carter, Luther J. 1979. “A New and Searching Look at NSF.” Science 204:1064–65.CrossRefGoogle Scholar
Dickersin, Kay. 1990. “The Existence of Publication Bias and Risk Factors for Its Occurrence.” Journal of the American Medical Association 263:1385–89.Google ScholarPubMed
Dickersin, Kay, Min, Yuan-I, and Meinert, Curtis L.. 1992. “Factors Influencing Publication of Research Results: Follow-Up of Applications Submitted to Two Institutional Review Boards.” Journal of the American Medical Association 267:374–79.Google ScholarPubMed
Easterbrook, Philippa J., Berlin, Jesse A., Gopalan, Ramana, and Matthews, David R.. 1991. “Publication Bias in Clinical Research.” Lancet 337:867–72.CrossRefGoogle ScholarPubMed
Espeland, Wendy Nelson, and Stevens, Mitchell L.. 1998. “Commensuration as a Social Process.” Annual Review of Sociology 24:313–43.CrossRefGoogle Scholar
Fanelli, Daniele. 2009. “How Many Scientists Fabricate and Falsify Research? A Systematic Review and Meta-Analysis of Survey Data.” PLOS ONE 4:1–11.CrossRefGoogle ScholarPubMed
Fanelli, Daniele 2010a. “Do Pressures to Publish Increase Scientists’ Bias? An Empirical Support from US States Data.” PLOS ONE 5:1–7.CrossRefGoogle Scholar
Fanelli, Daniele 2010b. “‘Positive’ Results Increase Down the Hierarchy of the Sciences.” PLOS ONE 5:1–10.CrossRefGoogle Scholar
Fanelli, Daniele 2012. “Negative Results Are Disappearing from Most Disciplines and Countries.” Scientometrics 90:891–904.CrossRefGoogle Scholar
Fang, Ferric C., Steen, R. Grant, and Casadevall, Arturo. 2012. “Misconduct Accounts for the Majority of Retracted Scientific Publications.” Proceedings of the National Academy of Sciences 109:17028–33.CrossRefGoogle ScholarPubMed
Fontanarosa, Phil B., and DeAngelis, Catherine D.. 2004. “Information for JAMA Authors—2004.” Journal of the American Medical Association 291:107–8.Google Scholar
Frank, Erica. 1996. “Editors’ Requests of Reviewers: A Study and a Proposal.” Preventative Medicine 25:102–4.CrossRefGoogle ScholarPubMed
Franzoni, Chiara, Scellato, Giuseppe, and Stephan, Paula. 2011. “Changing Incentives to Publish.” Science 333:702–3.CrossRefGoogle Scholar
Gillespie, Gilbert W., Chubin, Daryl E., and Kurzon, George M.. 1985. “Experience with NIH Peer Review: Researchers’ Cynicism and Desire for Change.” Science, Technology, and Human Values 10:44–54.CrossRefGoogle Scholar
Hargens, Lowell L., and Herting, Jerald R.. 1990. “A New Approach to Referees’ Assessments of Manuscripts.” Social Science Research 19:1–16.CrossRefGoogle Scholar
Harley, Diane, and Acord, Sophia Krzys. 2011. “Peer Review in Academic Promotion and Publishing: Its Meaning, Locus, and Future.” In The Future of Scholarly Communication Project. Berkeley: Center for Studies in Higher Education, UC Berkeley.Google Scholar
Haufe, Chris. 2013. “Why Do Funding Agencies Favor Hypothesis Testing?” Studies in History and Philosophy of Science 44:363–74.CrossRefGoogle Scholar
Ioannidis, John P. A. 1998. “Effect of the Statistical Significance of Results on the Time to Completion and Publication of Randomized Efficacy Trials.” Journal of the American Medical Association 279:281–86.Google ScholarPubMed
John, Leslie K., Loewenstein, George, and Prelec, Drazen. 2012. “Measuring the Prevalence of Questionable Research Practices with Incentives for Truth Telling.” Perspectives on Psychological Science 23:524–32.Google ScholarPubMed
Knowledge Ecology International. 2008. “Selected Innovation Prizes and Research Programs.” http://www.keionline.org/misc-docs/research_notes/kei_rn_2008_1.pdf.Google Scholar
Kolata, Gina. 2009. “Grant System Leads Cancer Researchers to Play It Safe.” New York Times, June 27. http://www.nytimes.com/2009/06/28/health/research/28cancer.html?pagewanted=all&_r=0.Google Scholar
Kuhn, Thomas S. 1977. “Objectivity, Value Judgment, and Theory Choice.” In The Essential Tension: Selected Studies in Scientific Tradition and Change, 320–39. Chicago: University of Chicago Press.CrossRefGoogle Scholar
Lal, Bhavya, Wilson, Alyson, Jonas, Seth, Lee, Elizabeth, Richards, Amy, and Peña, Vanessa. 2012. An Outcome Evaluation of the National Institutes of Health (NIH) Director’s Pioneer Award (NDPA) Program, FY 2004–2006. Alexandria, VA: Institute for Defense Analyses.Google Scholar
Lamont, Michèle. 2009. How Professors Think: Inside the Curious World of Academic Judgment. Cambridge, MA: Harvard University Press.CrossRefGoogle Scholar
Lancet. 2013. “Types of Article and Manuscript Requirements.” http://www.thelancet.com/lancet-information-for-authors/article-types-manuscript-requirements.Google Scholar
Langfeldt, Liv. 2001. “The Decision-Making Constraints and Processes of Grant Peer Review, and Their Effects on the Review Outcome.” Social Studies of Science 31:820–41.CrossRefGoogle Scholar
Lee, Carole J. 2012. “A Kuhnian Critique of Psychometric Research on Peer Review.” Philosophy of Science 79 (Proceedings): S859–S870.CrossRefGoogle Scholar
Lee, Carole J. 2013. “The Limited Effectiveness of Prestige as an Intervention on the Health of Medical Journal Publications.” Episteme 10:387–402.CrossRefGoogle Scholar
Lee, Carole J., and Erosheva, Elena A.. 2014. “Identifying Commensuration Bias in Grant Review.” NIH Peer Review Challenge, First Prize: Most Creative Idea. http://public.csr.nih.gov/Pages/challenge.aspx.Google Scholar
Lee, Carole J., and Schunn, Christian D.. 2011. “Social Biases and Solutions for Procedural Objectivity.” Hypatia: A Journal of Feminist Philosophy 26:352–73.CrossRefGoogle Scholar
Lee, Carole J., Sugimoto, Cassidy R., Zhang, Guo, and Cronin, Blaise. 2013. “Bias in Peer Review.” Journal of the American Society for Information Science 64:2–17.CrossRefGoogle Scholar
Luukkonen, Terttu. 2012. “Conservativism and Risk-Taking in Peer Review: Emerging ERC Practices.” Research Evaluation 21:48–60.CrossRefGoogle Scholar
Madden, Charles S., Easley, Richard W., and Dunn, Mark G.. 1995. “How Journal Editors View Replication Research.” Journal of Advertising 24:77–87.CrossRefGoogle Scholar
McCullough, Jim. 1989. “First Comprehensive Survey of NSF Applicants Focuses on Their Concerns about Proposal Review.” Science, Technology, and Human Values 14:78–88.CrossRefGoogle Scholar
McKinsey & Company. 2009. “And the Winner Is … ”: Capturing the Promise of Philanthropic Prizes. McKinsey & Company.Google Scholar
Murray, Christopher J. L., and Acharya, Arnab K.. 1997. “Understanding DALYs.” Journal of Health Economics 16:703–30.CrossRefGoogle ScholarPubMed
National Institutes of Health. 2008. “Enhancing Peer Review: The NIH Announces New Scoring Procedures for Evaluation of Research Applications Received for Potential FY2010 Funding.” NOT-OD-09-024. http://grants.nih.gov/grants/guide/notice-files/NOT-OD-09-024.html.Google Scholar
National Institutes of Health 2013. “About NIH: Mission.” http://nih.gov/about/mission.htm.Google Scholar
National Institutes of Health 2014a. “NIH Data Book.” http://report.nih.gov/nihdatabook/index.aspx.Google Scholar
National Institutes of Health 2014b. “NIH Grants Policy Statement.” http://grants.nih.gov/grants/policy/nihgps_2013/nihgps_2013.pdf.Google Scholar
National Institutes of Health 2014c. “NIH Reviewer Orientation.” http://grants.nih.gov/grants/peer/guidelines_general/reviewer_orientation.pdf.Google Scholar
National Research Council. 2007. Rising above the Gathering Storm: Energizing and Employing America for a Brighter Economic Future. Washington, DC: National Academies Press.Google Scholar
National Science Foundation. 2007a. “Enhancing Support of Transformative Research at the National Science Foundation.” NSB-07-32. http://www.nsf.gov/nsb/documents/2007/tr_report.pdf.Google Scholar
National Science Foundation 2007b. “Impact of Proposal and Award Management Mechanisms.” NSF 07-45. http://www.nsf.gov/pubs/2007/nsf0745/nsf0745.pdf.Google Scholar
National Science Foundation 2007c. “Important Notice No. 130: Transformative Research.” http://www.nsf.gov/pubs/2007/in130/in130.jsp.Google Scholar
National Science Foundation 2011a. “Empowering the Nation through Discovery and Innovation: NSF Strategic Plan for Fiscal Years (FY) 2011–2016.” NSF 11-047. http://www.nsf.gov/news/strategicplan/nsfstrategicplan_2011_2016.pdf.Google Scholar
National Science Foundation 2011b. National Science Foundation’s Merit Review Criteria: Review and Revisions. NSB/MR-11-22. http://www.nsf.gov/nsb/publications/2011/meritreviewcriteria.pdf.Google Scholar
National Science Foundation 2012. Proposal and Award Policies and Procedures Guide: Part I—Grant Proposal Guide. NSF 13-1. http://www.nsf.gov/pubs/policydocs/pappguide/nsf13001/gpgprint.pdf.Google Scholar
National Science Foundation 2013. “NSF at a Glance.” http://www.nsf.gov/about/glance.jsp.Google Scholar
Neuliep, James W., and Crandall, Rick. 1990. “Editorial Bias against Replication Research.” Journal of Social Behavior and Personality 5:85–90.Google Scholar
Neuliep, James W., and Crandall, Rick 1993. “Reviewer Bias against Replication Research.” Journal of Social Behavior and Personality 8:21–29.Google Scholar
Nosek, Brian A., and Bar-Anan, Yoav. 2012. “Scientific Utopia I: Opening Scientific Communication.” Psychological Inquiry: An International Journal for the Advancement of Psychological Theory 23:217–43.CrossRefGoogle Scholar
Nosek, Brian A., Spies, Jeffrey R., and Motyl, Matt. 2012. “Scientific Utoptia II: Restructuring Incentives and Practices to Promote Truth over Publishability.” Perspectives on Psychological Science 7:615–31.CrossRefGoogle ScholarPubMed
Olson, Carin M., Rennie, Drummond, Cook, Deborah, Dickersin, Kay, Flanagin, Annette, Hogan, Joseph W., Zhu, Qi, Reiling, Jennifer, and Pace, Brian. 2002. “Publication Bias in Editorial Decision Making.” Journal of the American Medical Association 287:2825–28.Google ScholarPubMed
Orszag, Peter R., and Holdren, John P.. 2010. “Science and Technology Priorities for the FY 2012 Budget, July 21, 2010.” Washington, DC: Office of Management and Budget. https://www.whitehouse.gov/sites/default/files/microsites/ostp/fy12-budget-guidance-memo.pdf.Google Scholar
Palmer, A. Richard. 2000. “Quasireplication and the Contract of Error: Lessons from Sex Ratios, Heritabilities and Fluctuating Asymmetry.” Annual Review of Ecology and Systematics 31:441–80.CrossRefGoogle Scholar
PeerJ. 2014. “Aims and Scope.” https://peerj.com/about/aims-and-scope/.Google Scholar
PLOS One. 2013. “PLOS ONE Journal Information.” http://www.plosone.org/static/information.Google Scholar
Rockey, Sally. 2011. “Correlation between Overall Impact Scores and Criterion Scores.” http://nexus.od.nih.gov/all/2011/03/08/overall-impact-and-criterion-scores/.Google Scholar
Salmon, Wesley C. 1990. “Rationality and Objectivity in Science or Tom Kuhn Meets Tom Bayes.” In Scientific Theories, ed. Savage, C. Wade, 175–204. Minneapolis: University of Minnesota Press.Google Scholar
Shatz, David. 2004. Peer Review: A Critical Inquiry. New York: Rowman & Littlefield.Google Scholar
Simes, Robert John. 1986. “Publication Bias: The Case for an International Registry of Clinical Trials.” Journal of Clinical Oncology 4:1529–41.CrossRefGoogle ScholarPubMed
Simmons, Joseph P., Nelson, Leif D., and Simonsohn, Uri. 2011. “False-Positive Psychology: Undisclosed Findings in Data Collection and Analysis Allows Presenting Anything as Significant.” Psychological Science 22:1359–66.CrossRefGoogle ScholarPubMed
Travis, G. D. L., and Collins, H. M.. 1991. “New Light on Old Boys: Cognitive and Institutional Particularism in the Peer Review System.” Science, Technology, and Human Values 16:322–41.CrossRefGoogle Scholar
Young, Neal S., Ioannidis, John P. A., and Al-Ubaydli, Omar. 2008. “Why Current Publication Practices May Distort Science.” PLOS Medicine 5:1418–22.CrossRefGoogle ScholarPubMed
Zollman, Kevin J. S. 2009. “Optimal Publishing Strategies.” Episteme 6:185–99.CrossRefGoogle Scholar