%0 Journal Article
%A Colquhoun, David
%T The Reproducibility Of Research And The Misinterpretation Of P Values
%D 2017
%R 10.1101/144337
%J bioRxiv
%P 144337
%X We wish to answer this question: If you observe a ‘significant’ P value after doing a single unbiased experiment, what is the probability that your result is a false positive? The weak evidence provided by P values between 0.01 and 0.05 is explored by exact calculations of false positive rates. When you observe P = 0.05, the odds in favour of there being a real effect (given by the likelihood ratio) are about 3. This is far weaker evidence than the odds of 19 to 1 that might, wrongly, be inferred from the P value. And if you want to limit the false positive rate to 5 percent, you would have to assume that you were 87% sure that there was a real effect before the experiment was done. If you observe P = 0.001, which gives likelihood ratio of 100:1 odds on there being a real effect, that would usually be regarded as conclusive, But the false event rate would still be 8% if the prior probability of a real effect was only 0.1. Despite decades of warnings, many areas of science still insist on labelling a result of P < 0.05 as ‘significant’. This must account for a substantial part of the lack of reproducibility in some areas of science. And this is before you get to the many other well-known problems, like multiple comparisons, lack of randomisation and P-hacking. Science is endangered by statistical misunderstanding, and by people who impose perverse incentives on scientists.
%U https://www.biorxiv.org/content/biorxiv/early/2017/06/04/144337.full.pdf