On the Predictability of Future Impact in Science

Abstract

Correctly assessing a scientist's past research impact and potential for future impact is key in recruitment decisions and other evaluation processes. While a candidate's future impact is the main concern for these decisions, most measures only quantify the impact of previous work. Recently, it has been argued that linear regression models are capable of predicting a scientist's future impact. By applying that future impact model to 762 careers drawn from three disciplines: physics, biology, and mathematics, we identify a number of subtle, but critical, flaws in current models. Specifically, cumulative non-decreasing measures like the h-index contain intrinsic autocorrelation, resulting in significant overestimation of their "predictive power". Moreover, the predictive power of these models depend heavily upon scientists' career age, producing least accurate estimates for young researchers. Our results place in doubt the suitability of such models, and indicate further investigation is required before they can be used in recruiting decisions.

Published version, 8 pages, 5 figures + Appendix

References in corpus (7)

• Does the h-index have predictive power?
• Universality of citation distributions: towards an objective measure of scientific impact
• The Possible Role of Resource Requirements and Academic Career-Choice Risk on Gender Differences in Publication Rate and Impact
• Persistence and Uncertainty in the Academic Career
• The role of mentorship in protege performance
• How relevant is the predictive power of the h-index? A case study of the time-dependent Hirsch index
• The case for caution in predicting scientists' future impact

Cited by in corpus (17)

• The Matthew effect in empirical data
• Ranking in evolving complex networks
• The h-index is no longer an effective correlate of scientific reputation
• Quantifying the impact of weak, strong, and super ties in scientific careers
• Author Impact Factor: tracking the dynamics of individual scientific impact
• Growing complex network of citations of scientific papers -- measurements and modeling
• Inheritance patterns in citation networks reveal scientific memes
• Quantifying and predicting success in show business
• The Distribution of the Asymptotic Number of Citations to Sets of Publications by a Researcher or From an Academic Department Are Consistent With a Discrete Lognormal Model
• Predictability of extreme events in social media
• Evaluating the impact of interdisciplinary research: a multilayer network approach
• Prediction Methods and Applications in the Science of Science: A Survey
• The Z-index: A geometric representation of productivity and impact which accounts for information in the entire rank-citation profile
• A variant of the h-index to measure recent performance
• Individual and gender inequality in computer science: A career study of cohorts from 1970 to 2000
• Non-cumulative measures of researcher citation impact
• Collective Attention towards Scientists and Research Topics