Uncertain research country rankings. Should we continue producing uncertain rankings?
arXiv:2312.17560 · doi:10.2478/jdis-2025-0030
Abstract
Purpose: Citation-based assessments of countries' research capabilities often misrepresent their ability to achieve breakthrough advancements. These assessments commonly classify Japan as a developing country, which contradicts its prominent scientific standing. The purpose of this study is to investigate the underlying causes of such inaccurate assessments and to propose methods for conducting more reliable evaluations. Design/methodology/approach: The study evaluates the effectiveness of top-percentile citation metrics as indicators of breakthrough research. Using case studies of selected countries and research topics, the study examines how deviations from lognormal citation distributions impact the accuracy of these percentile indicators. A similar analysis is conducted using university data from the Leiden Ranking to investigate citation distribution deviations at the institutional level. Findings: The study finds that inflated lower tails in citation distributions lead to undervaluation of research capabilities in advanced technological countries, as captured by some percentile indicators. Conversely, research-intensive universities exhibit the opposite trend: a reduced lower tail relative to the upper tail, which causes percentile indicators to overestimate their actual research capacity. Research limitations: The descriptions are mathematical facts that are self-evident. Practical implications: Due to variations in citation patterns across countries and institutions, the Ptop 10%/P and Ptop 1%/P ratios are not universal predictors of breakthrough research. Evaluations should move away from these metrics. Relying on inappropriate citation-based measures could lead to poor decision-making in research policy, undermining the effectiveness of research strategies and their outcomes.
arXiv comment29 pages, 6 figures, 4 tables
References in corpus (12)
- An index to quantify an individual's scientific research output
- A Review of Theory and Practice in Scientometrics
- Universality of citation distributions: towards an objective measure of scientific impact
- A review of the characteristics of 108 author-level bibliometric indicators
- On the calculation of percentile-based bibliometric indicators
- Science Quality and the Value of Inventions
- Systematic analysis of agreement between metrics and peer review in the UK REF
- In which fields are citations indicators of research quality?
- Are there too many uncited articles? Zero inflated variants of the discretised lognormal and hooked power law distributions
- National peer-review research assessment exercises for the hard sciences can be a complete waste of money: the Italian case
- The inconsistency of h-index: a mathematical analysis
- Like-for-like bibliometric substitutes for peer review: advantages and limits of indicators calculated from the ep index