From rankings to funnel plots: the question of accounting for uncertainty when measuring university research performance

Abstract

The work applies the funnel plot methodology to measure and visualize uncertainty in the research performance of Italian universities in the science disciplines. The performance assessment is carried out at both discipline and overall university level. The findings reveal that for most universities the citation-based indicator used gives insufficient statistical evidence to infer that their research productivity is inferior or superior to the average. This general observation is one that we could indeed expect in a higher education system that is essentially non-competitive. The question is whether the introduction of uncertainty in performance reporting, while technically sound, could weaken institutional motivation to work towards continuous improvement.

References in corpus (11)

• How do you define and measure research productivity?
• A heuristic approach to author name disambiguation in bibliometrics databases for large-scale research assessments
• Assessing the varying level of impact measurement accuracy as a function of the citation window length
• The importance of accounting for the number of co-authors and their order when assessing research performance at the individual level in the life sciences
• A farewell to the MNCS and like size-independent indicators
• What is the appropriate length of the publication period over which to assess research performance?
• Revisiting size effects in higher education research productivity
• The dispersion of research performance within and between universities as a potential indicator of the competitive intensity in higher education systems
• A national-scale cross-time analysis of university research performance
• National research assessment exercises: a measure of the distortion of performance rankings when labor input is treated as uniform
• Funnel plots for visualizing uncertainty in the research performance of institutions

Cited by in corpus (3)

• The ratio of top scientists to the academic staff as an indicator of the competitive strength of universities
• Conceptual difficulties in the use of statistical inference in citation analysis
• The Research Production of Nations and Departments: A Statistical Model for the Share of Publications