The VQR, Italy's second national research assessment: Methodological failures and ranking distortions

Abstract

The 2004-2010 VQR, completed in July 2013, was Italy's second national research assessment exercise. The VQR performance evaluation followed a pattern also seen in other nations, in being based on a selected subset of products. In this work we identify the exercise's methodological weaknesses and measure the distortions that result from them in the university performance rankings. First we create a scenario in which we assume the efficient selection of the products to be submitted by the universities and from this simulate a set of rankings applying the precise VQR rating criteria. Next we compare these "VQR rankings" with those that would derive from application of more appropriate bibliometrics. Finally we extend the comparison to university rankings based on the entire scientific production for the period, as indexed in the Web of Science.

References in corpus (12)

• How do you define and measure research productivity?
• Evaluating research: from informed peer review to bibliometrics
• A heuristic approach to author name disambiguation in bibliometrics databases for large-scale research assessments
• Allocative efficiency in public research funding:can bibliometrics help?
• National research assessment exercises: a comparison of peer review and bibliometrics rankings
• The importance of accounting for the number of co-authors and their order when assessing research performance at the individual level in the life sciences
• Revisiting the scaling of citations for research assessment
• Citations versus journal impact factor as proxy of quality: Could the latter ever be preferable?
• The role of information asymmetry in the market for university-industry research collaboration
• National peer-review research assessment exercises for the hard sciences can be a complete waste of money: the Italian case
• Inefficiency in selecting products for submission to national research assessment exercises
• Peer review research assessment: a sensitivity analysis of performance rankings to the share of research product evaluated

Cited by in corpus (5)

• The north-south divide in the Italian higher education system
• On tit for tat: Franceschini and Maisano versus ANVUR regarding the Italian research assessment exercise VQR 2011-2014
• A methodology to measure the effectiveness of academic recruitment and turnover
• Errors and secret data in the Italian research assessment exercise. A comment to a reply
• Metrics and peer review agreement at the institutional level