Gradient Descent Quantizes ReLU Network Features

Abstract

Deep neural networks are often trained in the over-parametrized regime (i.e. with far more parameters than training examples), and understanding why the training converges to solutions that generalize remains an open problem. Several studies have highlighted the fact that the training procedure, i.e. mini-batch Stochastic Gradient Descent (SGD) leads to solutions that have specific properties in the loss landscape. However, even with plain Gradient Descent (GD) the solutions found in the over-parametrized regime are pretty good and this phenomenon is poorly understood. We propose an analysis of this behavior for feedforward networks with a ReLU activation function under the assumption of small initialization and learning rate and uncover a quantization effect: The weight vectors tend to concentrate at a small number of directions determined by the input data. As a consequence, we show that for given input data there are only finitely many, "simple" functions that can be obtained, independent of the network size. This puts these functions in analogy to linear interpolations (for given input data there are finitely many triangulations, which each determine a function by linear interpolation). We ask whether this analogy extends to the generalization properties - while the usual distribution-independent generalization property does not hold, it could be that for e.g. smooth functions with bounded second derivative an approximation property holds which could "explain" generalization of networks (of unbounded size) to unseen inputs.

References in corpus (3)

• Opening the Black Box of Deep Neural Networks via Information
• Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapes
• Norm-Based Capacity Control in Neural Networks

Cited by in corpus (15)

• How Neural Networks Extrapolate: From Feedforward to Graph Neural Networks
• Gradient Dynamics of Shallow Univariate ReLU Networks
• Convergence analysis for gradient flows in the training of artificial neural networks with ReLU activation
• Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like process
• Phase diagram for two-layer ReLU neural networks at infinite-width limit
• Revealing the Structure of Deep Neural Networks via Convex Duality
• Convex Geometry and Duality of Over-parameterized Neural Networks
• Towards Understanding the Condensation of Neural Networks at Initial Training
• On Sparsity in Overparametrised Shallow ReLU Networks
• How Implicit Regularization of ReLU Neural Networks Characterizes the Learned Function -- Part I: the 1-D Case of Two Layers with Random First Layer
• The Convex Geometry of Backpropagation: Neural Network Gradient Flows Converge to Extreme Points of the Dual Convex Program
• Initializing ReLU networks in an expressive subspace of weights
• Implicit Convex Regularizers of CNN Architectures: Convex Optimization of Two- and Three-Layer Networks in Polynomial Time
• Embedding Principle: a hierarchical structure of loss landscape of deep neural networks
• Mean-field Analysis of Piecewise Linear Solutions for Wide ReLU Networks