Evaluating research and researchers by the journal impact factor: is it better than coin flipping?
arXiv:1809.10999 · doi:10.1016/j.joi.2019.01.009
Abstract
The journal impact factor (JIF) is the average of the number of citations of the papers published in a journal, calculated according to a specific formula; it is extensively used for the evaluation of research and researchers. The method assumes that all papers in a journal have the same scientific merit, which is measured by the JIF of the publishing journal. This implies that the number of citations measures scientific merits but the JIF does not evaluate each individual paper by its own number of citations. Therefore, in the comparative evaluation of two papers, the use of the JIF implies a risk of failure, which occurs when a paper in the journal with the lower JIF is compared to another with fewer citations in the journal with the higher JIF. To quantify this risk of failure, this study calculates the failure probabilities, taking advantage of the lognormal distribution of citations. In two journals whose JIFs are ten-fold different, the failure probability is low. However, in most cases when two papers are compared, the JIFs of the journals are not so different. Then, the failure probability can be close to 0.5, which is equivalent to evaluating by coin flipping.
arXiv comment24 pages, 1 figure and 4 tables in the main text, plus 1 table as supplementary material. One single pdf file
References in corpus (5)
- Core Elements in the Process of Citing Publications: A Conceptual Overview of the Literature
- Citations versus journal impact factor as proxy of quality: Could the latter ever be preferable?
- Systematic analysis of agreement between metrics and peer review in the UK REF
- Citation count distributions for large monodisciplinary journals
- Probability and expected frequency of breakthroughs - a robust method of research assessment based on the double rank property of citation distributions
Cited by in corpus (3)
- Like-for-like bibliometric substitutes for peer review: advantages and limits of indicators calculated from the ep index
- A critical review of the proposed reforms of the academic performance indicators applied in the assessment of researchers' performance in Hungary
- Validated research assessment based on highly cited researchers