Abstract
Purpose: To determine whether papers contributed by National Academy of Sciences (NAS) members perform differently than direct submissions.
Data/Methods: 55,889 original papers published in PNAS from 1997 through 2014. Regression analysis measuring total citations, controlling for editorial track (Contributed, Direct, Communicated), date of publication, and paper topic.
Main findings: Contributed papers consistently underperformed against Direct submissions, receiving 9% fewer citations, ceteris paribus. The effect was greatest for Social Sciences papers (12% fewer citations). Nonetheless, the main effect has attenuated over the past decade, from 13.6% fewer citations in 2005 to just 2.2% fewer citations in 2014.
Significance: Successive editorial policies placing limits, restrictions, and other qualifications on the publication privileges of NAS members may be responsible for the submission of better performing Contributed papers.
Background
Over the years, successive editorial policies for the Proceedings of the National Academy of Sciences (PNAS) have placed limits, restrictions, and other qualifications on the publication privileges of NAS members. The intentions of these policy changes were to improve upon the transparency of the editorial process and external review, and secondly, to encourage the submission of higher-quality papers. Should these policies have their intended effects, we should expect to observe a gradual improvement in the performance of papers contributed by NAS authors compared to direct submissions.
PNAS currently allows two manuscript submission tracks1: Direct, in which papers are subject to single-blind peer review; and Contributed, in which a NAS member may secure their own reviewers and submit their comments alongside the paper. A third submission track (Communicated), in which papers are sent directly to a NAS member, who is responsible for overseeing the peer review process, was abolished in 2010.2
Example changes to PNAS publication policies
1972 – NAS members can Contribute or Communicate up to 10 papers per year.
1989–1992. Quota reduced to 6 papers/yr.
1993–1995. Quota reduced to 5 papers/yr.
1994 – July. Conflict of Interest (COI) disclaimer is added to NAS members’ paper.
1995 – Dec. COI disclaimer requirement is removed.
1996 – Quota reduced to 4 papers/yr.
2001 – Names of reviewers are required when Contributing or Communicating papers.
2006 – Jan. NAS member must be a corresponding author on Contributed papers.
2006 – Aug. PNAS begins listing author contributions on papers.
2007 – May. Two reviewers are required for Contributed papers
2008 – April. NAS members with a significant COI with their Contributed papers are required to use Direct submission track.
2010 – July. PNAS eliminates Communicated submissions.
2014 – COI decision of 2008 repealed.
2014 – April. Voluntarily listing reviewer names on contributed papers.
2015 – Oct. Requiring reviewer names on contributed papers.
Data and methods
The dataset includes 55,889 original research papers published in PNAS from 1997 through 2014, years for which submission track information was available from the publisher. Perspectives, Commentary, and other non-research papers were excluded from the dataset. For each PNAS paper, the complete citation record was extracted from the Web of Science (Thomson Reuters) during the first week of November 2015 and matched with the submission track (Direct, Contributed, Communicated) and journal section (e.g. Biological Sciences—Microbiology) using the paper’s Digital Object Identifier (DOI) or by matching the paper’s volume and page number.
Our primary objective was to evaluate the performance of papers by submission track. To arrive at a relative performance measure, we compared the performance of Contributed and Communicated papers against Direct Submission papers for each year of publication. Performance was measured by the total number of citations to a PNAS paper. As citation distributions are highly skewed, we normalized the distribution by adding one citation to each paper before taking the natural logarithm of the result.3
Our primary analysis involved building a linear regression model, using log(total citations) as the response variable, and submission track (Contributed, Communicated) as two indicator variables. We controlled for publication date within each year by using the journal’s issue number. Finally, we controlled for the topic of each paper by using both the primary domain of each paper (Biological Sciences, Physical Sciences, Social Sciences), as well as its specific field (e.g. Microbiology). As some fields are classified under multiple domains—for example, Anthropology papers are classified under both Biological Science and Social Sciences—we nested the field variable within the domain variable.
Results
PNAS grew by 44% over the observation period, from 2,437 research papers published in 1997 to 3,521 papers in 2014 (Table 1), during which, the proportion of Direct submissions increased from one-in-five (22%) PNAS papers to over three-quarters (77%). In contrast, Contributed submissions have remained relatively constant. In 2010, PNAS eliminated the Communicated submission track.
In every year, from 1997 to 2014, Contributed papers underperformed against Direct submissions (Figure 1, Table 2). Controlling for the date of publication and journal section (field of study), Contributed papers received 9% fewer citations, on average, than Direct submissions. The effect was greatest for Social Sciences papers (12% fewer citations). Nonetheless, the main effect has attenuated over the past decade, from 13.6% fewer citations in 2005 to just 2.2% fewer citations in 2014.
For Communicated papers, the results were mixed. In half of the years, Communicated papers performed better than Direct Submissions; in the other half, they performed worse.
Inaugural papers (typically the first Contributed paper published by a NAS member) received 8% more citations (95% C.I. 4% to 12%, p=0.0001) than other Contributed papers and performed statistically no different than Direct submissions (citation deviance = +2%, 95% C.I. -1% to 6%, p=0.170).
After their second calendar year of publication, 3.8% (526 of 13866) of Contributed papers remained uncited, compared to 2.6% (789 of 30790) of Direct submissions. Uncitedness varied by PNAS field (Figure 2, Table 3). Controlling for date of publication and journal section, Contributed papers were nearly twice as likely to remain uncited (Odds Ratio = 1.7, 95% C.I. 1.5 to 1.9, Chi2<.0001). Uncitedness among Contributed papers, however, has been declining at 3% per annum (95% C.I. -5% to -1%, p=0.01).
Percentage of uncited papers by section in their first two calendar years of publication (Blue=Direct, Red=Contributed). Numbers and percentages are found in Table 3.
In spite of their general underperformance, the 10% most-cited Contributed papers outperformed the top 10% of Direct submissions, receiving 8% more citations, on average (95% C.I. 5% to 11%, p=<.0001) over the entire dataset (Figure 3, Table 4). Consistent with our main findings, these effects have also been attenuating over the past decade and were not detectable in the last five years of publication (Table 4).
Histogram of the 10% most-cited PNAS papers for each year (1997-2014) by submission track. Regression results for each year of publication are found in Table 4.
Prior Work
Studying PNAS papers published between June 1, 2004 and April 26, 2005 and using a similar regression model, Rand and Pfeiffer4 reported that Contributed papers received 10% fewer citations compared to Direct submissions, a result that is consistent with our findings for those years (see Table 2). Similarly, they reported that the 10% most-cited Contributed papers outperformed the 10% most-cited Direct submissions, consistent with our findings for 2004 and 2005 (see Table 4). Aldhous5 reported that that only a very small number of NAS members used the Contributed track frequently; the majority of members published less than one contributed paper per year, on average.
Study Limitations
This study compared the performance of papers by submission track not by NAS member status. While all Contributed papers must include at least one NAS author, it is not known whether Direct submissions included any NAS authors. Having the membership status of each author would have permitted us to disentangle author effects from submission track effects.
Conclusions
The difference in citation performance between Contributed and Direct submissions has been declining over the past decade, suggesting that successive editorial policies intended to improve upon the transparency and submission of higher-quality contributed papers is the primary explanation.
Competing Interests
This study was conducted as part of a consulting project for the publishers of PNAS. The author (PD) was responsible entirely for the design of the study, gathering and analysis of the data, and was the sole author of this report.
Footnotes
↵1 PNAS Editorial Policies. http://www.pnas.org/site/authors/guidelines.xhtml
↵2 Schekman, R. PNAS will eliminate Communicated submissions in July 2010. http://dx.doi.org/10.1073/pnas.0909515106
↵3 Adding one citation to each paper allows for papers that have not yet been cited to be included in the analysis as the log of zero is a logical impossibility. While this procedure right-shifts the distribution, our analysis focuses on the comparative performance of tracks and not their absolute values.
↵4 Rand DG & Pfeiffer T (2009) Systematic Differences in Impact across Publication Tracks at PNAS. PLoS ONE 4(12):e8092. http://dx.doi.org/10.1371/journal.pone.0008092
↵5 Aldhous P (2014) Scientific publishing: The inside track. Nature 510:330-332. http://dx.doi.org/10.1038/510330a
