Revisiting the scaling of citations for research assessment

Abstract

Over the past decade, national research evaluation exercises, traditionally conducted using the peer review method, have begun opening to bibliometric indicators. The citations received by a publication are assumed as proxy for its quality, but they require standardization prior to use in comparative evaluation of organizations or individual scientists: the citation data must be standardized, due to the varying citation behavior across research fields. The objective of this paper is to compare the effectiveness of the different methods of normalizing citations, in order to provide useful indications to research assessment practitioners. Simulating a typical national research assessment exercise, he analysis is conducted for all subject categories in the hard sciences and is based on the Thomson Reuters Science Citation Index-Expanded. Comparisons show that the citations average is the most effective scaling parameter, when the average is based only on the publications actually cited.

References in corpus (4)

• Universality of citation distributions: towards an objective measure of scientific impact
• Evaluating research: from informed peer review to bibliometrics
• Nonuniversal power law scaling in the probability distribution of scientific citations
• A national-scale cross-time analysis of university research performance

Cited by in corpus (34)

• How do you define and measure research productivity?
• Predicting long-term publication impact through a combination of early citations and journal impact factor
• The relationship between the number of authors of a publication, its citations and the impact factor of the publishing journal: Evidence from Italy
• Gender differences in research performance within and between countries: Italy vs Norway
• Are the authors of highly cited articles also the most productive ones?
• Do interdisciplinary research teams deliver higher gains to science?
• The VQR, Italy's second national research assessment: Methodological failures and ranking distortions
• Individual research performance: a proposal for comparing apples to oranges
• Should the research performance of scientists be distinguished by gender?
• Gender bias in academic recruitment
• The north-south divide in the Italian higher education system
• The effect of a country's name in the title of a publication on its visibility and citability
• Ranking research institutions by the number of highly-cited articles per scientist
• Evaluating university research: same performance indicator, different rankings
• Does your surname affect the citability of your publications?
• How long do top scientists maintain their stardom? An analysis by region, gender and discipline: evidence from Italy
• The ratio of top scientists to the academic staff as an indicator of the competitive strength of universities
• Diversification versus specialization in scientific research: which strategy pays off?
• A nation's foreign and domestic professors: which have better research performance? (The Italian case)
• An investigation on the skewness patterns and fractal nature of research productivity distributions at field and discipline level
• Are all citations worth the same? Valuing citations by the value of the citing items
• A multivariate stochastic model to assess research performance
• Selection committees for academic recruitment: does gender matter?
• The determinants of academic career advancement: evidence from Italy
• A methodology to measure the effectiveness of academic recruitment and turnover
• How important is choice of the scaling factor in standardizing citations?
• Testing for universality of Mendeley readership distributions
• A comparison of university performance scores and ranks by MNCS and FSS
• Funnel plots for visualizing uncertainty in the research performance of institutions
• A new approach to measure the scientific strengths of territories
• The dispersion of the citation distribution of top scientists' publications
• A methodology to compute the territorial productivity of scientists: The case of Italy
• Informed peer review for publication assessments: Are improved impact measures worth the hassle?
• A systematic empirical comparison of different approaches for normalizing citation impact indicators