Predicting Neural Network Accuracy from Weights

Abstract

We show experimentally that the accuracy of a trained neural network can be predicted surprisingly well by looking only at its weights, without evaluating it on input data. We motivate this task and introduce a formal setting for it. Even when using simple statistics of the weights, the predictors are able to rank neural networks by their performance with very high accuracy (R2 score more than 0.98). Furthermore, the predictors are able to rank networks trained on different, unobserved datasets and with different architectures. We release a collection of 120k convolutional neural networks trained on four different datasets to encourage further research in this area, with the goal of understanding network training and performance better.

arXiv commentUpdated the Small CNN Zoo dataset: reduced the maximal learning rate and got rid of multiple bad runs. Replaced all the experiments with the new numbers. Added MLP. Fixed typo in the abstract (R2 score instead of Kendall's tau). Added several earlier related works to the literature overview

References in corpus (7)

• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Predicting the Generalization Gap in Deep Networks with Margin Distributions
• Freeze-Thaw Bayesian Optimization
• Are All Layers Created Equal?
• Norm-Based Capacity Control in Neural Networks
• Towards Task and Architecture-Independent Generalization Gap Predictors
• Classifying the classifier: dissecting the weight space of neural networks

Cited by in corpus (10)

• Predicting trends in the quality of state-of-the-art neural networks without access to training or testing data
• Deep Learning Through the Lens of Example Difficulty
• Perceptron Theory Can Predict the Accuracy of Neural Networks
• Understanding and Accelerating Neural Architecture Search with Training-Free and Theory-Grounded Metrics
• Neural Complexity Measures
• Increasing Trustworthiness of Deep Neural Networks via Accuracy Monitoring
• What Does Rotation Prediction Tell Us about Classifier Accuracy under Varying Testing Environments?
• Practical Assessment of Generalization Performance Robustness for Deep Networks via Contrastive Examples
• Parameter-Based Value Functions
• ICLR 2021 Challenge for Computational Geometry & Topology: Design and Results