Skip to main content
bioRxiv
  • Home
  • About
  • Submit
  • ALERTS / RSS
Advanced Search
New Results

The reproducibility of research and the misinterpretation of P values

View ORCID ProfileDavid Colquhoun
doi: https://doi.org/10.1101/144337
David Colquhoun
University College London
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for David Colquhoun
  • Abstract
  • Full Text
  • Info/History
  • Metrics
  • Preview PDF
Loading

Abstract

We wish to answer this question If you observe a “significant” P value after doing a single unbiased < experiment, what is the probability that your result is a false positive?. The weak evidence provided by P values between 0.01 and 0.05 is explored by exact calculations of false positive rates.

When you observe P = 0.05, the odds in favour of there being a real effect (given by the likelihood ratio) are about 3:1. This is far weaker evidence than the odds of 19 to 1 that might, wrongly, be inferred from the P value. And if you want to limit the false positive rate to 5 %, you would have to assume that you were 87% sure that there was a real effect before the experiment was done.

If you observe P = 0.001 in a well-powered experiment, it gives a likelihood ratio of almost 100:1 odds on there being a real effect. That would usually be regarded as conclusive, But the false positive rate would still be 8% if the prior probability of a real effect was only 0.1. And, in this case, if you wanted to achieve a false positive rate of 5% you would need to observe P = 0.00045.

It is recommended that the terms “significant” and “non-significant” should never be used. Rather, P values should be supplemented by specifying the prior probability that would be needed to produce a specified (e.g. 5%) false positive rate. It may also be helpful to specify the minimum false positive rate associated with the observed P value.

Despite decades of warnings, many areas of science still insist on labelling a result of P < 0.05 as “significant”. This practice must account for a substantial part of the lack of reproducibility in some areas of science. And this is before you get to the many other well-known problems, like multiple comparisons, lack of randomisation and P-hacking. Science is endangered by statistical misunderstanding, and by university presidents and research funders who impose perverse incentives on scientists.

Footnotes

  • Email: d.colquhoun{at}ucl.ac.uk

Copyright 
The copyright holder for this preprint is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under a CC-BY-NC 4.0 International license.
Back to top
PreviousNext
Posted July 24, 2017.
Download PDF
Email

Thank you for your interest in spreading the word about bioRxiv.

NOTE: Your email address is requested solely to identify you as the sender of this article.

Enter multiple addresses on separate lines or separate them with commas.
The reproducibility of research and the misinterpretation of P values
(Your Name) has forwarded a page to you from bioRxiv
(Your Name) thought you would like to see this page from the bioRxiv website.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
Share
The reproducibility of research and the misinterpretation of P values
David Colquhoun
bioRxiv 144337; doi: https://doi.org/10.1101/144337
Digg logo Reddit logo Twitter logo CiteULike logo Facebook logo Google logo Mendeley logo
Citation Tools
The reproducibility of research and the misinterpretation of P values
David Colquhoun
bioRxiv 144337; doi: https://doi.org/10.1101/144337

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero
  • Tweet Widget
  • Facebook Like
  • Google Plus One

Subject Area

  • Scientific Communication and Education
Subject Areas
All Articles
  • Animal Behavior and Cognition (2420)
  • Biochemistry (4779)
  • Bioengineering (3319)
  • Bioinformatics (14637)
  • Biophysics (6620)
  • Cancer Biology (5158)
  • Cell Biology (7405)
  • Clinical Trials (138)
  • Developmental Biology (4343)
  • Ecology (6864)
  • Epidemiology (2057)
  • Evolutionary Biology (9893)
  • Genetics (7331)
  • Genomics (9502)
  • Immunology (4540)
  • Microbiology (12644)
  • Molecular Biology (4928)
  • Neuroscience (28245)
  • Paleontology (199)
  • Pathology (803)
  • Pharmacology and Toxicology (1384)
  • Physiology (2014)
  • Plant Biology (4482)
  • Scientific Communication and Education (975)
  • Synthetic Biology (1295)
  • Systems Biology (3907)
  • Zoology (722)