Perceptron Theory Can Predict the Accuracy of Neural Networks

Abstract

Multilayer neural networks set the current state of the art for many technical classification problems. But, these networks are still, essentially, black boxes in terms of analyzing them and predicting their performance. Here, we develop a statistical theory for the one-layer perceptron and show that it can predict performances of a surprisingly large variety of neural networks with different architectures. A general theory of classification with perceptrons is developed by generalizing an existing theory for analyzing reservoir computing models and connectionist models for symbolic reasoning known as vector symbolic architectures. Our statistical theory offers three formulas leveraging the signal statistics with increasing detail. The formulas are analytically intractable, but can be evaluated numerically. The description level that captures maximum details requires stochastic sampling methods. Depending on the network model, the simpler formulas already yield high prediction accuracy. The quality of the theory predictions is assessed in three experimental settings, a memorization task for echo state networks (ESNs) from reservoir computing literature, a collection of classification datasets for shallow randomly connected networks, and the ImageNet dataset for deep convolutional neural networks. We find that the second description level of the perceptron theory can predict the performance of types of ESNs, which could not be described previously. The theory can predict deep multilayer neural networks by being applied to their output layer. While other methods for prediction of neural networks performance commonly require to train an estimator model, the proposed theory requires only the first two moments of the distribution of the postsynaptic sums in the output neurons. The perceptron theory compares favorably to other methods that do not rely on training an estimator model.

16 pages, 14 figures

References in corpus (17)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Methods for Interpreting and Understanding Deep Neural Networks
• On the Opportunities and Risks of Foundation Models
• Shortcut Learning in Deep Neural Networks
• FitNets: Hints for Thin Deep Nets
• Prevalence of Neural Collapse during the terminal phase of deep learning training
• Classification using Hyperdimensional Computing: A Review
• A Survey on Hyperdimensional Computing aka Vector Symbolic Architectures, Part II: Applications, Cognitive Models, and Challenges
• A Survey on Hyperdimensional Computing aka Vector Symbolic Architectures, Part I: Models and Data Transformations
• Peephole: Predicting Network Performance Before Training
• Vector Symbolic Architectures as a Computing Framework for Emerging Hardware
• Predicting trends in the quality of state-of-the-art neural networks without access to training or testing data
• Zero-Cost Proxies for Lightweight NAS
• Predicting Neural Network Accuracy from Weights
• Efficient Decoding of Compositional Structure in Holistic Representations
• Computing on Functions Using Randomized Vector Representations
• Computing with Hypervectors for Efficient Speaker Identification

Cited by in corpus (2)

• Efficient Decoding of Compositional Structure in Holistic Representations
• On Effects of Compression with Hyperdimensional Computing in Distributed Randomized Neural Networks