Confidence intervals for normalised citation counts: Can they delimit underlying research capability?
arXiv:1710.08708 · doi:10.1016/j.joi.2017.09.002
Abstract
Normalised citation counts are routinely used to assess the average impact of research groups or nations. There is controversy over whether confidence intervals for them are theoretically valid or practically useful. In response, this article introduces the concept of a group's underlying research capability to produce impactful research. It then investigates whether confidence intervals could delimit the underlying capability of a group in practice. From 123120 confidence interval comparisons for the average citation impact of the national outputs of ten countries within 36 individual large monodisciplinary journals, moderately fewer than 95% of subsequent indicator values fall within 95% confidence intervals from prior years, with the percentage declining over time. This is consistent with confidence intervals effectively delimiting the research capability of a group, although it does not prove that this is the cause of the results. The results are unaffected by whether internationally collaborative articles are included.
arXiv commentIt is reasonable to use confidence intervals to estimate underlying research capability and prediction intervals to estimate future research output, although system changes over time mean that they should be used cautiously
References in corpus (9)
- Characterizing and modeling citation dynamics
- Three practical field normalised alternative indicator formulae for research evaluation
- Caveats for using statistical significance tests in research assessments
- Universality of Performance Indicators based on Citation and Reference Counts
- Distributions for cited articles from individual subjects and years
- Citation count distributions for large monodisciplinary journals
- A multivariate stochastic model to assess research performance
- The Accuracy of Confidence Intervals for Field Normalised Indicators
- Conceptual difficulties in the use of statistical inference in citation analysis