Co-Citation Percentile Rank and JYUcite: a new network-standardized output-level citation influence metric and its implementation using Dimensions API

Judging value of scholarly outputs quantitatively remains a difficult but unavoidable challenge. Most of the proposed solutions suffer from three fundamental shortcomings: they involve i) the concept of journal, in one way or another, ii) calculating arithmetic averages from extremely skewed distributions, and iii) binning data by calendar year. Here, we introduce a new metric Co-citation Percentile Rank (CPR), that relates the current citation rate of the target output taken at resolution of days since first citable, to the distribution of current citation rates of outputs in its co-citation set, as its percentile rank in that set. We explore some of its properties with an example dataset of all scholarly outputs from University of Jyväskylä spanning multiple years and disciplines. We also demonstrate how CPR can be efficiently implemented with Dimensions database API, and provide a publicly available web resource JYUcite, allowing anyone to retrieve CPR value for any output that has a DOI and is indexed in the Dimensions database. Finally, we discuss how CPR remedies failures of the Relative Citation Ratio (RCR), and remaining issues in situations where CPR too could potentially lead to biased judgement of value.

unable to find co-citation set metadata. The quartiles and average of the citation rates, and the 156 size of the co-citation set are also calculated and saved for illustrative purpose. 176 Metadata for total of 41 713 outputs from JYU current research system published between  size, number of co-citations for which times_cited metadata was present, co-citation median 186 citation rate, quartiles and average, the CPR metric, and the solve time (seconds it took to retrieve the co-citation set and calculate the metrics) as well as UNIX timestamp of the time 188 when the calculation was done. 189 Out of those, 13 170 had at least 10 co-citations for which metadata could be found, and these 190 were retained for beta regression analysis.

191
The example dataset is is publicly available at https://gitlab.jyu.fi/jyucite/published_cpr and 192 published in Seppänen (2020) 193 194 Statistical analysis 195 Descriptive analyses on the effect of number of citers on CPR solving time and size of the co-196 citation set were done with quantile regression (Koenker 2020). Quantile regression model 197 fits are estimated using the R1 statistic (Koenker & Machado 1999, Long 2020 Because the distribution of CPR data is by definition bound to unit interval (0,100), and, as is 199 typical for any citation metric, also heteroscedastic and asymmetric, we model its response to  After fitting the overall beta regression of CPR to citation rate, we next explore the 211 differences in that relationship between academic disciplines. Each output in the database is 212 affiliated with one or more academic departments at University of Jyväskylä (JYU). For 213 purposes of this analysis, we simplify the data by merging data from units falling under same 214 discipline (e.g. Institute for Education Research is merged with Department of Education).

215
For outputs still having multiple affiliations after the mergers, the record is replicated so that 216 an output occurs once for each affiliated department. Departments having fewer than 100 outputs were excluded from the final analysis. The derived expanded dataset contains 13871 218 observations from 16 different academic departments (Table 1.) 219 We then model CPR as a function of citation rate and academic department, including 220 interaction terms and a precision parameter accounting for heteroscedasticity along the 221 citation rate range.

222
The R code to replicate the analyses and figures presented here is publicly available at 223 https://gitlab.jyu.fi/jyucite/published_cpr and published in Seppänen (2020)

229
With our computing setup (a single RHEL7.7 Server, 1 x 2.60GHz CPU, 2GB RAM, PHP 230 5.4.16, in university's fiber-optic network), solving CPR for an output was within reasonable 231 time performance and solve time was linearly scaled with number of citers, though variance 232 was considerable (Fig. 2).  Fig. 3. Log-log plot of the co-citation set size as function of times a peer-reviewed output has 251 been cited. Dot opacity is set at 0.1 in R plotting function so that overlap darkness illustrates 252 density of observations. Quantile regression for median (line) and 90% prediction interval 253 (shaded area) and 99% confidence interval (dashed lines). Co-citation set size ~ 39.73 * times 254 cited + 13.82 (N = 13337, model fit: R¹(0.5) = 0.54). The extreme high outliers result from an 255 output being cited by reviews and books, or large reference works, which alone can bring 256 hundreds or thousands of entries, respectively, to output's co-citation set.
Overall beta regression of CPR as a function of citation rate illustrates that though CPR 259 increases with increasing citation rate, the relationship has considerable variation, as should 260 be expected if CPR captures sub-field specific citation influence. Some outputs achieve CPR 261 around 50 (i.e. are cited at least as frequently as half of their co-citations) when they get cited 262 1-2 times per 365 days, while other outputs need 9-10 citations per 365 days to achieve 263 similar CPR (Fig 4.).

292
History & Ethnology on the other hand consistently has significantly higher intercept term 293 then other departments, i.e. CPR for a History & Ethnology output tends to be higher, given a 294 certain citation rate. departments at JYU. Dot opacity is set at 0.1 in R plotting function so that overlap darkness illustrates density of observations. Beta regression 299 predicted mean (line) and 90% prediction interval (shaded area). Note that x-axis is logarithmic.
The contrast between academic disciplines found are consistent with conventional wisdom 303 and highlight the utility of CPR as a field-normalized citation rate metric. In Mathematics & 304 Statistics, publication volumes are relatively low both individually and collectively, and 305 works often cite just a few foundational references. In Physics, the three particle accelerators 306 and nanoscale materials research at JYU result in outputs that are in very large, fast-moving 307 fields where massively collaborative works get cited quickly. In History & Ethnology, the 308 discipline has smaller and slower publication volumes as many outputs are monographs, and 309 perhaps citations behaviours are also more siloed to fine-grained sub-fields and by language.

310
It should be noted though, that the sample size for History and Ethnology is small here, and 311 furthermore that sparse sample shows more variation than other disciplines (Fig 4.), so 312 inference must be cautious 313 Implemetation using Dimensions API 314 The efficiency of the Dimensions API is remarkable for this task: for an ouvre of M target hence large contributions to co-citation sets -than typical articles.

336
Faster accrual of the co-citation set using Dimensions also means our implementation of CPR 337 is not very vulnerable to finite number effects. For median, typical target output, mere three 338 citers bring its co-citation set size above 100, where percentile can be confidently stated as 339 integer value without having to interpolate between observed values.  size N and quartiles of citation rates in the co-citation set (Fig 5.) JYUcite's own database and is younger than 100 days, it is returned immediately without new 430 calculation and does not count in the daily rate limit. It would begin to gain (proportionally large numbers if the other field typically has longer 441 length of reference lists) relatively frequently cited co-citations into its co-citation set, 442 repressing its CPR. Thus interdisciplinary influence, which typically is seen as a merit, may 443 actually erode the CPR value of an output (see also Waltman 2015). On the other hand, it 444 could also be argued that once an output begins to become relevant in another discipline, it 445 should be start to get compared to outputs in that field, otherwise it would appear unduly 446 influential compared to those.  Sciences (relatively few of the thousands of co-citations accrued are from outside fields), it outputs that end up cited in reference works outside their own field are cited extensively in 469 their old fields before that, and tend to be aged. The example article above has, at time of 470 writing this, been cited 301 times and has a co-citation set of 21.616 outputs, so contribution 471 of the encyclopaedia is at most 20% of the co-citation set.  (2008)", cited more than 68.000 times in total, or over 5000 times per every 365 days).

488
Is inclusion of such "boilerplate" general methodology outputs in the comparison set a source 489 of bias? They are always far more frequently cited than a typical target output, and thus 490 outrank them, lowering the target output's CPR. On the other hand, they often objectively do 491 have more significant influence on research than any field-specific research output. Also, 492 they cannot be automatically excluded using either blacklists or citation rate thresholds, as the 493 target output may well be directly in the same field as, or a co-citation have citation rate 494 rivalling, e.g. a statistical methodology "boilerplate" reference.

495
But even where hyper-cited co-citations are seen as a source of bias for CPR, the magnitude 496 of such bias decreases rapidly as target output gains more citers and thereby median of 40 497 new co-citations per new citer (Fig 3.). Single or few hyper-cited co-citations in a large set do not have a large effect on the percentile rank, while they would easily distort a metric relating 499 the target to averages in the set.

Delayed recognition and instant fading 501
The value of a scholarly output is not necessarily present or recognized immediately upon 502 publication. Some other, later discovery may suddenly reveal an important aspect of an 503 earlier output, or its relevance in another field of research may be get re-discovered 504 serendipitously, awakening the output to suddenly increased influence, many years after first is not necessarily clear: should an awakened sleeping beauty be considered more valuable 520 than a new discovery, or should we acknowledge that CPR reflects whatever it was that kept 521 the sleeping beauty from gaining influence sooner? Notably, the direction of change in CPR