Predicting trends in the quality of state-of-the-art neural networks without access to training or testing data

Abstract

In many applications, one works with neural network models trained by someone else. For such pretrained models, one may not have access to training data or test data. Moreover, one may not know details about the model, e.g., the specifics of the training data, the loss function, the hyperparameter values, etc. Given one or many pretrained models, it is a challenge to say anything about the expected performance or quality of the models. Here, we address this challenge by providing a detailed meta-analysis of hundreds of publicly-available pretrained models. We examine norm based capacity control metrics as well as power law based metrics from the recently-developed Theory of Heavy-Tailed Self Regularization. We find that norm based metrics correlate well with reported test accuracies for well-trained models, but that they often cannot distinguish well-trained versus poorly-trained models. We also find that power law based metrics can do much better -- quantitatively better at discriminating among series of well-trained models with a given architecture; and qualitatively better at discriminating well-trained versus poorly-trained models. These methods can be used to identify when a pretrained neural network has problems that cannot be detected simply by examining training/test accuracies.

35 pages, 8 tables, 17 figures. To appear in Nature Communications

References in corpus (8)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Power-law distributions in empirical data
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• Cleaning large correlation matrices: tools from random matrix theory
• Spectrally-normalized margin bounds for neural networks
• Norm-Based Capacity Control in Neural Networks
• Predicting Neural Network Accuracy from Weights
• Multiplicative noise and heavy tails in stochastic optimization

Cited by in corpus (13)

• Dimension of activity in random neural networks
• Multiplicative noise and heavy tails in stochastic optimization
• Perceptron Theory Can Predict the Accuracy of Neural Networks
• Extended Anderson Criticality in Heavy-Tailed Neural Networks
• Random matrix analysis of deep neural network weight matrices
• Taxonomizing local versus global structure in neural network loss landscapes
• Hessian Eigenspectra of More Realistic Nonlinear Models
• Sparse Quantized Spectral Clustering
• Learning from learning machines: a new generation of AI technology to meet the needs of science
• Unmasking the physical information inherent to interstellar spectral line profiles with Machine Learning. I. Application of LTE to HCN and HNC transitions
• Post-mortem on a deep learning contest: a Simpson's paradox and the complementary roles of scale metrics versus shape metrics
• Boundary between noise and information applied to filtering neural network weight matrices
• Fast Unsupervised Deep Outlier Model Selection with Hypernetworks