Polynomial Regression As an Alternative to Neural Nets

Abstract

Despite the success of neural networks (NNs), there is still a concern among many over their "black box" nature. Why do they work? Here we present a simple analytic argument that NNs are in fact essentially polynomial regression models. This view will have various implications for NNs, e.g. providing an explanation for why convergence problems arise in NNs, and it gives rough guidance on avoiding overfitting. In addition, we use this phenomenon to predict and confirm a multicollinearity property of NNs not previously reported in the literature. Most importantly, given this loose correspondence, one may choose to routinely use polynomial models instead of NNs, thus avoiding some major problems of the latter, such as having to set many tuning parameters and dealing with convergence issues. We present a number of empirical results; in each case, the accuracy of the polynomial approach matches or exceeds that of NN approaches. A many-featured, open-source software package, polyreg, is available.

arXiv comment23 pages, 1 figure, 13 tables

References in corpus (3)

• Opening the Black Box of Deep Neural Networks via Information
• Forward Thinking: Building Deep Random Forests
• Neural networks and rational functions

Cited by in corpus (9)

• Uncertainty in Neural Networks: Approximately Bayesian Ensembling
• Towards a mathematical framework to inform Neural Network modelling via Polynomial Regression
• NN2Poly: A polynomial representation for deep feed-forward artificial neural networks
• Avoiding Spurious Local Minima in Deep Quadratic Networks
• Bayesian Neural Network Ensembles
• Dropout as a Regularizer of Interaction Effects
• Variations on the Chebyshev-Lagrange Activation Function
• Rational activation functions in neural networks with uniform based loss functions and its application in classification
• Recoding latent sentence representations -- Dynamic gradient-based activation modification in RNNs