Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Perspective
  • Published:

Association of percentile ranking with citation impact and productivity in a large cohort of de novo NIMH-funded R01 grants

Molecular Psychiatry volume 20, pages 1030–1036 (2015)Cite this article

Subjects

Abstract

Previous reports from National Institutes of Health and National Science Foundation have suggested that peer review scores of funded grants bear no association with grant citation impact and productivity. This lack of association, if true, may be particularly concerning during times of increasing competition for increasingly limited funds. We analyzed the citation impact and productivity for 1755 de novo investigator-initiated R01 grants funded for at least 2 years by National Institute of Mental Health between 2000 and 2009. Consistent with previous reports, we found no association between grant percentile ranking and subsequent productivity and citation impact, even after accounting for subject categories, years of publication, duration and amounts of funding, as well as a number of investigator-specific measures. Prior investigator funding and academic productivity were moderately strong predictors of grant citation impact.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Figure 1
Figure 2
Figure 3
Figure 4

Similar content being viewed by others

References

  1. Demicheli V, Di Pietrantonj C . Peer review for improving the quality of grant applications. Cochrane Database Syst Rev 2007; MR000003.

  2. Bornmann L, Daniel H-D . Reliability, fairness, and predictive validity of committee peer review. BIF Futur 2004; 19: 7–19.

    Google Scholar 

  3. Danthi N, Wu CO, Shi P, Lauer M . Percentile ranking and citation impact of a large cohort of National Heart, Lung, and Blood Institute-funded cardiovascular R01 grants. Circ Res 2014; 114: 600–606.

    Article  CAS  Google Scholar 

  4. Kaltman JR, Evans FJ, Danthi NS, Wu CO, DiMichele DM, Lauer MS . Prior publication productivity, grant percentile ranking, and topic-normalized citation impact of NHLBI cardiovascular R01 grants. Circ Res 2014; 115: 617–624.

    Article  CAS  Google Scholar 

  5. Berg JM . Science policy: well-funded investigators should receive extra scrutiny. Nature 2012; 489: 203.

    Article  CAS  Google Scholar 

  6. Scheiner SM, Bouchie LM . The predictive power of NSF reviewers and panels. Front Ecol Environ 2013; 11: 406–407.

    Article  Google Scholar 

  7. Gallo SA, Carpenter AS, Irwin D, McPartland CD, Travis J, Reynders S et al. The validation of peer review through research impact measures and the implications for funding strategies. PLoS One 2014; 9: e106474.

    Article  Google Scholar 

  8. Bornmann L, Marx W . How good is research really? Measuring the citation impact of publications with percentiles increases correct assessments and fair comparisons. EMBO Rep 2013; 14: 226–230.

    Article  CAS  Google Scholar 

  9. Pendlebury D White Paper: Using Biobliometrics in Evaluating Research (available athttp://wokinfo.com/media/mtrp/UsingBibliometricsinEval_WP.pdf2008.

  10. Breiman L . Statistical modeling: the two cultures. Stat Sci 2001; 16: 199–215.

    Article  Google Scholar 

  11. Gorodeski EZ, Ishwaran H, Kogalur UB, Blackstone EH, Hsich E, Zhang ZM et al. Use of hundreds of electrocardiographic biomarkers for prediction of mortality in postmenopausal women the Women’s Health Initiative. Circ Qual Outcomes 2011; 4: 521–532.

    Article  Google Scholar 

  12. Wickham H ggplot2: elegant graphics for data analysis 2009; http://had.co.nz/ggplot2/book.

  13. Ishwaran H, Kogalur UB Random Forests for Survival, Regression and Classification (RF-SRC)2015; http://cran.r-project.org/web/packages/randomForestSRC/.

  14. Jacob BA, Lefgren L . The impact of research grant funding on scientific productivity. J Pub Econ 2011; 95: 1168–1177.

    Article  Google Scholar 

  15. Carey B . Blazing trails in brain science. New York Times, 2014; http://www.nytimes.com/2014/02/04/science/blazing-trails-in-brain-science.html?_r=0(accessed on 22 March 2015).

    Google Scholar 

  16. Clavería LE, Guallar E, Camí J, Conde J, Pastor R, Ricoy JR et al. Does peer review predict the performance of research projects in health sciences? Scientometrics 2000; 47: 11–23.

    Article  Google Scholar 

Download references

Acknowledgements

Lisa Alberts, Dr Thomas Insel and the NIMH Division Directors for comments on earlier versions of the analysis and manuscript. All authors are full-time employees of the US Department of Health and Human Services and conducted this work as part of their official federal duties.

Disclaimer

The views expressed in this paper are those of the authors and do not necessarily represent those of the NHLBI, NIMH, NIH or the US Department of Health and Human Services.

Author information

Authors and Affiliations

  1. Office of Science Policy, Planning, and Communications, National Institute of Mental Health (NIMH), Bethesda, MD, USA

    J M Doyle, K Quinn & Y A Bodenstein

  2. Division of Cardiovascular Sciences, Office of Biostatistics Research, National Heart, Lung and Blood Institute (NHLBI), Bethesda, MD, USA

    C O Wu

  3. Division of Cardiovascular Sciences of the National Heart, Advanced Technologies and Surgery Branch, Lung, and Blood Institute (NHLBI), Bethesda, MD, USA

    N Danthi

  4. Division of Cardiovascular Sciences, Office of the Director, National Heart, Lung and Blood Institute (NHLBI), Bethesda, MD, USA

    M S Lauer

Authors
  1. J M Doyle
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. K Quinn
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Y A Bodenstein
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. C O Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. N Danthi
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. M S Lauer
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to M S Lauer.

Ethics declarations

Competing interests

The authors declare no conflict of interest.

Additional information

Supplementary Information accompanies the paper on the Molecular Psychiatry website

Supplementary information

PowerPoint slides

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Doyle, J., Quinn, K., Bodenstein, Y. et al. Association of percentile ranking with citation impact and productivity in a large cohort of de novo NIMH-funded R01 grants. Mol Psychiatry 20, 1030–1036 (2015). https://doi.org/10.1038/mp.2015.71

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/mp.2015.71

This article is cited by

Search

Advanced search

Quick links