Publication and other reporting biases in cognitive sciences: detection, prevalence, and prevention

doi:10.1016/j.tics.2014.02.010

Trends in Cognitive Sciences

Volume 18, Issue 5, May 2014, Pages 235-241

https://doi.org/10.1016/j.tics.2014.02.010 Get rights and content

Highlights

•
Publication and other reporting biases can have major detrimental effects on the credibility and value of research evidence.
•
There is substantial empirical evidence of publication and reporting biases in cognitive science disciplines.
•
Common types of bias are discussed and potential solutions are proposed.

Recent systematic reviews and empirical evaluations of the cognitive sciences literature suggest that publication and other reporting biases are prevalent across diverse domains of cognitive science. In this review, we summarize the various forms of publication and reporting biases and other questionable research practices, and overview the available methods for probing into their existence. We discuss the available empirical evidence for the presence of such biases across the neuroimaging, animal, other preclinical, psychological, clinical trials, and genetics literature in the cognitive sciences. We also highlight emerging solutions (from study design to data analyses and reporting) to prevent bias and improve the fidelity in the field of cognitive science research.

Introduction

The promise of science is that the generation and evaluation of evidence is done transparently to promote an efficient, self-correcting process. However, multiple biases may produce inefficiency in knowledge building. In this review, we discuss the importance of publication and other reporting biases, and present some potential correctives that may reduce bias without disrupting innovation. Consideration of publication and other reporting biases is particularly timely for cognitive sciences because it is a field that is expanding rapidly. As such, preventing or remedying these biases will have substantial impact on the efficient development of a credible corpus of published research.

Section snippets

Definitions of biases and relevance for cognitive sciences

The terms ‘publication bias’ and ‘selective reporting bias’ refer to the differential choice to publish studies or report particular results, respectively, depending on the nature or directionality of findings [1]. There are several forms of such biases in the literature [2], including: (i) study publication bias, where studies are less likely to be published when they reach nonstatistically significant findings; (ii) selective outcome reporting bias, where multiple outcomes are evaluated in a

Tests for single studies, specific topics, and wider disciplines

Explicit documentation of publication and other reporting biases requires availability of protocols, data, and results of primary analyses from conducted studies so that these can be compared against the published literature. However, these are not often available. A few empirical studies have retrieved study protocols from authors or trial data from submissions to the US Food and Drug Administration (FDA) 6, 7, 8, 9. These studies have shown that deviations in the analysis plan between

Neuroimaging

With over 3000 publications annually in the past decade, including several meta-analyses [27] (Figure 1), the identification of potential publication or reporting biases is of crucial interest for the future of neuroimaging. The first study to evaluate evidence for an excess of statistically significant results in neuroimaging focused on brain volumetric studies based on region of interests (ROIs) analyses of psychiatric conditions [28]. The study analyzed 41 meta-analyses (461 data sets) and

Approaches to prevent bias

Most scientists embrace fundamental scientific values, such as disinterestedness and transparency, even while believing that other scientists do not [60]. However, noble intentions may not be sufficient to decrease biases when people are not able to recognize or control their own biases 61, 62, rationalize biases through motivated reasoning [63], and are embedded in a culture that implicitly rewards the expression of biases for personal career advancement (i.e., publication over accuracy).

Concluding remarks

Overall, publication and other selective reporting biases are probably prevalent and influential in diverse cognitive sciences. Different approaches have been proposed to remedy these biases. Box 2 summarizes some outstanding questions in this process. As shown in Figure 2, there are multiple entry points for each of diverse ‘problems’ from designing to conducting studies, analyzing data, and reporting results. ‘Solutions’ are existing, but uncommon, practices that should improve the fidelity

Acknowledgments

S.P.D. acknowledges funding from the National Institute on Drug Abuse US Public Health Service grant DA017441 and a research stipend from the Institute of Medicine (IOM) (Puffer/American Board of Family Medicine/IOM Anniversary Fellowship). M.R.M. acknowledges support for his time from the UK Centre for Tobacco Control Studies (a UKCRC Public Health Research Centre of Excellence) and funding from British Heart Foundation, Cancer Research UK, the Economic and Social Research Council, the Medical

References (96)

A. Sayo
Study factors influencing ventricular enlargement in schizophrenia: a 20 year follow-up meta-analysis
Neuroimage
(2012)
Z. Jonasson
Meta-analysis of sex differences in rodent models of learning and memory: a review of behavioral and biological data
Neurosci. Biobehav. Rev.
(2005)
E. Sena
How can we improve the pre-clinical development of drugs for stroke?
Trends Neurosci.
(2007)
M.R. Munafò et al.
Dissecting the genetic architecture of human personality
Trends Cogn. Sci.
(2011)
E. Evangelou
Perceived information gain from randomized trials correlates with publication in high-impact factor journals
J. Clin. Epidemiol.
(2012)
C.D. Chambers
Registered reports: a new publishing initiative at Cortex
Cortex
(2013)
R.A. Poldrack
Guidelines for reporting an fMRI study
Neuroimage
(2008)
J. Carp
The secret lives of experiments: methods reporting in the fMRI literature
Neuroimage
(2012)
R. Rosenthal
The file drawer problem and tolerance for null results
Psychol. Bull.
(1979)
J.P. Ioannidis et al.
Early extreme contradictory estimates may appear in published research: the Proteus phenomenon in molecular genetics research and randomized trials
J. Clin. Epidemiol.
(2005)

S. Green

Cochrane Handbook for Systematic Reviews of Interventions, version 5.1.0

(2011)

K. Dwan

Systematic review of the empirical evidence of study publication bias and outcome reporting bias

PLoS ONE

(2008)

G. Vogel

Scientific misconduct. Psychologist accused of fraud on ‘astonishing scale’

Science

(2011)

G. Vogel

Scientific misconduct. Fraud charges cast doubt on claims of DNA damage from cell phone fields

Science

(2008)

G. Vogel

Developmental biology. Fraud investigation clouds paper on early cell fate

Science

(2006)

N. Saquib

Practices and impact of primary outcome adjustment in randomized controlled trials: meta-epidemiologic study

BMJ

(2013)

E.H. Turner

Publication bias in antipsychotic trials: an analysis of efficacy comparing the published literature to the US Food and Drug Administration database

PLoS Med.

(2012)

E.H. Turner

Selective publication of antidepressant trials and its influence on apparent efficacy

N. Engl. J. Med.

(2008)

K. Dwan

Systematic review of the empirical evidence of study publication bias and outcome reporting bias – an updated review

PLoS ONE

(2013)

J.A. Sterne

Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials

BMJ

(2011)

J. Lau

The case of the misleading funnel plot

BMJ

(2006)

J.L. Peters

Comparison of two methods to detect publication bias in meta-analysis

JAMA

(2006)

R.M. Harbord

A modified test for small-study effects in meta-analyses of controlled trials with binary endpoints

Stat. Med.

(2006)

J.B. Copas et al.

A robust P-value for treatment effect in meta-analysis with publication bias

Stat. Med.

(2008)

J. Copas et al.

A bound for publication bias based on the fraction of unpublished studies

Biometrics

(2004)

L.V. Hedges et al.

Estimating effect size under publication bias: small sample properties and robustness of a random effects selection model

J. Educ. Behav. Stat.

(1996)

T. Pfeiffer

Quantifying selective reporting and the Proteus phenomenon for multiple datasets with similar bias

PLoS ONE

(2011)

G.L. Gadbury et al.

Inappropriate fiddling with statistical analyses to obtain a desirable p-value: tests to detect its presence in published literature

PLoS ONE

(2012)

J.P. Ioannidis et al.

An exploratory test for an excess of significant findings

Clin. Trials

(2007)

J.P.A. Ioannidis

Clarifications on the application and interpretation of the test for excess significance and its extensions

J. Math. Psychol.

(2014)

G. Francis

The psychology of replication and replication in psychology

Perspect. Psychol. Sci.

(2012)

S.P. David

Potential reporting bias in FMRI studies of the brain

PLoS ONE

(2013)

G. Rücker

Detecting and adjusting for small-study effects in meta-analysis

Biom. J.

(2011)

S. Duval et al.

Trim and fill: a simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis

Biometrics

(2000)

J.P. Ioannidis

Adjusting for bias: a user's guide to performing plastic surgery on meta-analyses of observational studies

Int. J. Epidemiol.

(2011)

S. Thompson

A proposed method of bias adjustment for meta-analyses of published observational studies

Int. J. Epidemiol.

(2011)

R.G. Jennings et al.

Publication bias in neuroimaging research: implications for meta-analyses

Neuroinformatics

(2012)

J.P. Ioannidis

Excess significance bias in the literature on brain volume abnormalities

Arch. Gen. Psychiatry

(2011)

S. Borgwardt

Why are psychiatric imaging methods clinically unreliable? Conclusions and practical guidelines for authors, editors and reviewers

Behav. Brain Funct.

(2012)

P. Fusar-Poli

Evidence of reporting biases in voxel-based morphometry (VBM) studies of psychiatric and neurological disorders

Hum. Brain Mapp.

(2013)

C.G. Begley et al.

Drug development: raise standards for preclinical cancer research

Nature

(2012)

F. Prinz

Believe it or not: how much can we rely on published data on potential drug targets?

Nat. Rev. Drug Discov.

(2011)

G. ter Riet

Publication bias in laboratory animal research: a survey on magnitude, drivers, consequences and potential solutions

PLoS ONE

(2012)

E.S. Sena

Publication bias in reports of animal stroke studies leads to major overstatement of efficacy

PLoS Biol.

(2010)

J.P. Ioannidis

Extrapolating from animals to humans

Sci. Transl. Med.

(2012)

M.R. Macleod

Evidence for the efficacy of NXY-059 in experimental focal cerebral ischaemia is confounded by study quality

Stroke

(2008)

K.S. Button

Power failure: why small sample size undermines the reliability of neuroscience

Nat. Rev. Neurosci.

(2013)

T.D. Sterling

Publication decisions and their possible effects on inferences drawn from tests of significance, or vice versa

J. Am. Stat. Assoc.

(1959)

Cited by (331)

Registered reports and replications: An ongoing Journal of School Psychology initiative
2024, Journal of School Psychology
Recent psychological research suggests that many published studies cannot be replicated (e.g., Open Science Collaboration, 2015). The inability to replicate results suggests that there are influences and biases in the publication process that encourage publication of unusual—rather than representative—results, and that also discourage independent replication of published studies. A brief discussion of the ways in which publication bias and professional incentives may distort the research literature in school psychology is contrasted against the importance of replications and preregistration of research (i.e., registered reports) as self-correcting mechanisms for research in school psychology. The limitations of current practices, coupled with the importance of registered reports and replications as self-correcting mechanisms, provide the context for this ongoing initiative in the Journal of School Psychology. Processes for manuscript submission, review, and publication are presented to encourage researchers to preregister studies and submit replications for publication.
Neural correlates of mindfulness meditation and hypnosis on magnetic resonance imaging: similarities and differences. A scoping review
2024, Journal of Neuroradiology
Mindfulness meditation (MM) and hypnosis practices are gaining interest in mental health, but their physiological mechanisms remain poorly understood. This study aimed to synthesize the functional, morphometric and metabolic changes associated with each practice using magnetic resonance imaging (MRI), and to identify their similarities and differences.
MRI studies investigating MM and hypnosis in mental health, specifically stress, anxiety, and depression, were systematically screened following PRISMA guidelines from four research databases (PubMed, Web of Science, Embase, PsycINFO) between 2010 and 2022.
In total, 97 references met the inclusion criteria (84 for MM and 13 for hypnosis). This review showed common and divergent points regarding the regions involved and associated brain connectivity during MM practice and hypnosis. The primary commonality between mindfulness and hypnosis was decreased default mode network intrinsic activity and increased central executive network - salience network connectivity. Increased connectivity between the default mode network and the salience network was observed in meditative practice and mindfulness predisposition, but not in hypnosis.
While MRI studies provide a better understanding of the neural basis of hypnosis and meditation, this review underscores the need for more rigorous studies.
Success versus failure in cognitive control: Meta-analytic evidence from neuroimaging studies on error processing
2024, Neuroscience and Biobehavioral Reviews
Brain mechanisms of error processing have often been investigated using response interference tasks and focusing on the posterior medial frontal cortex, which is also implicated in resolving response conflict in general. Thereby, the role other brain regions may play has remained undervalued. Here, activation likelihood estimation meta-analyses were used to synthesize the neuroimaging literature on brain activity related to committing errors versus responding successfully in interference tasks and to test for commonalities and differences. The salience network and the temporoparietal junction were commonly recruited irrespective of whether responses were correct or incorrect, pointing towards a general involvement in coping with situations that call for increased cognitive control. The dorsal posterior cingulate cortex, posterior thalamus, and left superior frontal gyrus showed error-specific convergence, which underscores their consistent involvement when performance goals are not met. In contrast, successful responding revealed stronger convergence in the dorsal attention network and lateral prefrontal regions. Underrecruiting these regions in error trials may reflect failures in activating the task-appropriate stimulus-response contingencies necessary for successful response execution.
The roles, challenges, and merits of the p value
2023, Patterns
Since the 18th century, the p value has been an important part of hypothesis-based scientific investigation. As statistical and data science engines accelerate, questions emerge: to what extent are scientific discoveries based on p values reliable and reproducible? Should one adjust the significance level or find alternatives for the p value? Inspired by these questions and everlasting attempts to address them, here, we provide a systematic examination of the p value from its roles and merits to its misuses and misinterpretations. For the latter, we summarize modest recommendations to handle them. In parallel, we present the Bayesian alternatives for seeking evidence and discuss the pooling of p values from multiple studies and datasets. Overall, we argue that the p value and hypothesis testing form a useful probabilistic decision-making mechanism, facilitating causal inference, feature selection, and predictive modeling, but that the interpretation of the p value must be contextual, considering the scientific question, experimental design, and statistical principles.
Spindle-dependent memory consolidation in healthy adults: A meta-analysis
2023, Neuropsychologia
Accumulating evidence suggests a central role for sleep spindles in the consolidation of new memories. However, no meta-analysis of the association between sleep spindles and memory performance has been conducted so far. Here, we report meta-analytical evidence for spindle-memory associations and investigate how multiple factors, including memory type, spindle type, spindle characteristics, and EEG topography affect this relationship. The literature search yielded 53 studies reporting 1427 effect sizes, resulting in a small to moderate effect for the average association. We further found that spindle-memory associations were significantly stronger for procedural memory than for declarative memory. Neither spindle types nor EEG scalp topography had an impact on the strength of the spindle-memory relation, but we observed a distinct functional role of global and fast sleep spindles, especially for procedural memory. We also found a moderation effect of spindle characteristics, with power showing the largest effect sizes. Collectively, our findings suggest that sleep spindles are involved in learning, thereby representing a general physiological mechanism for memory consolidation.
Reproducibility in Neuroimaging Analysis: Challenges and Solutions
2023, Biological Psychiatry: Cognitive Neuroscience and Neuroimaging
Recent years have marked a renaissance in efforts to increase research reproducibility in psychology, neuroscience, and related fields. Reproducibility is the cornerstone of a solid foundation of fundamental research—one that will support new theories built on valid findings and technological innovation that works. The increased focus on reproducibility has made the barriers to it increasingly apparent, along with the development of new tools and practices to overcome these barriers. Here, we review challenges, solutions, and emerging best practices with a particular emphasis on neuroimaging studies. We distinguish 3 main types of reproducibility, discussing each in turn. Analytical reproducibility is the ability to reproduce findings using the same data and methods. Replicability is the ability to find an effect in new datasets, using the same or similar methods. Finally, robustness to analytical variability refers to the ability to identify a finding consistently across variation in methods. The incorporation of these tools and practices will result in more reproducible, replicable, and robust psychological and brain research and a stronger scientific foundation across fields of inquiry.

View all citing articles on Scopus

View full text

Trends in Cognitive Sciences

ReviewPublication and other reporting biases in cognitive sciences: detection, prevalence, and prevention

Highlights

Introduction

Section snippets

Definitions of biases and relevance for cognitive sciences

Tests for single studies, specific topics, and wider disciplines

Neuroimaging

Approaches to prevent bias

Concluding remarks

Acknowledgments

Neuroimage

Neurosci. Biobehav. Rev.

Trends Neurosci.

Trends Cogn. Sci.

J. Clin. Epidemiol.

Cortex

Neuroimage

Neuroimage

Psychol. Bull.

J. Clin. Epidemiol.

Cochrane Handbook for Systematic Reviews of Interventions, version 5.1.0

Systematic review of the empirical evidence of study publication bias and outcome reporting bias

PLoS ONE

Scientific misconduct. Psychologist accused of fraud on ‘astonishing scale’

Science

Scientific misconduct. Fraud charges cast doubt on claims of DNA damage from cell phone fields

Science

Developmental biology. Fraud investigation clouds paper on early cell fate

Science

Practices and impact of primary outcome adjustment in randomized controlled trials: meta-epidemiologic study

BMJ

Publication bias in antipsychotic trials: an analysis of efficacy comparing the published literature to the US Food and Drug Administration database

PLoS Med.

Selective publication of antidepressant trials and its influence on apparent efficacy

N. Engl. J. Med.

Systematic review of the empirical evidence of study publication bias and outcome reporting bias – an updated review

PLoS ONE

Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials

BMJ

The case of the misleading funnel plot

BMJ

Comparison of two methods to detect publication bias in meta-analysis

JAMA

A modified test for small-study effects in meta-analyses of controlled trials with binary endpoints

Stat. Med.

A robust P-value for treatment effect in meta-analysis with publication bias

Stat. Med.

A bound for publication bias based on the fraction of unpublished studies

Biometrics

Estimating effect size under publication bias: small sample properties and robustness of a random effects selection model

J. Educ. Behav. Stat.

Quantifying selective reporting and the Proteus phenomenon for multiple datasets with similar bias

PLoS ONE

Inappropriate fiddling with statistical analyses to obtain a desirable p-value: tests to detect its presence in published literature

PLoS ONE

An exploratory test for an excess of significant findings

Clin. Trials

Clarifications on the application and interpretation of the test for excess significance and its extensions

J. Math. Psychol.

The psychology of replication and replication in psychology

Perspect. Psychol. Sci.

Potential reporting bias in FMRI studies of the brain

PLoS ONE

Detecting and adjusting for small-study effects in meta-analysis

Biom. J.

Trim and fill: a simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis

Biometrics

Adjusting for bias: a user's guide to performing plastic surgery on meta-analyses of observational studies

Int. J. Epidemiol.

A proposed method of bias adjustment for meta-analyses of published observational studies

Int. J. Epidemiol.

Publication bias in neuroimaging research: implications for meta-analyses

Neuroinformatics

Excess significance bias in the literature on brain volume abnormalities

Arch. Gen. Psychiatry

Why are psychiatric imaging methods clinically unreliable? Conclusions and practical guidelines for authors, editors and reviewers

Behav. Brain Funct.

Evidence of reporting biases in voxel-based morphometry (VBM) studies of psychiatric and neurological disorders

Hum. Brain Mapp.

Review
Publication and other reporting biases in cognitive sciences: detection, prevalence, and prevention