Universality of Gradient Descent Neural Network Training

Abstract

It has been observed that design choices of neural networks are often crucial for their successful optimization. In this article, we therefore discuss the question if it is always possible to redesign a neural network so that it trains well with gradient descent. This yields the following universality result: If, for a given network, there is any algorithm that can find good network weights for a classification task, then there exists an extension of this network that reproduces these weights and the corresponding forward output by mere gradient descent training. The construction is not intended for practical computations, but it provides some orientation on the possibilities of meta-learning and related approaches.

References in corpus (12)

• Sequence to Sequence Learning with Neural Networks
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
• How transferable are features in deep neural networks?
• The Loss Surfaces of Multilayer Networks
• Large-Scale Evolution of Image Classifiers
• Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
• No bad local minima: Data independent training error guarantees for multilayer neural networks
• Spurious Local Minima are Common in Two-Layer ReLU Neural Networks
• Approximating Continuous Functions by ReLU Nets of Minimal Width
• Meta-Learning and Universality: Deep Representations and Gradient Descent can Approximate any Learning Algorithm
• Biologically inspired alternatives to backpropagation through time for learning in recurrent neural nets