Gradient Dynamics of Shallow Univariate ReLU Networks

Abstract

We present a theoretical and empirical study of the gradient dynamics of overparameterized shallow ReLU networks with one-dimensional input, solving least-squares interpolation. We show that the gradient dynamics of such networks are determined by the gradient flow in a non-redundant parameterization of the network function. We examine the principal qualitative features of this gradient flow. In particular, we determine conditions for two learning regimes:kernel and adaptive, which depend both on the relative magnitude of initialization of weights in different layers and the asymptotic behavior of initialization coefficients in the limit of large network widths. We show that learning in the kernel regime yields smooth interpolants, minimizing curvature, and reduces to cubic splines for uniform initializations. Learning in the adaptive regime favors instead linear splines, where knots cluster adaptively at the sample points.

References in corpus (15)

• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• A Mean Field View of the Landscape of Two-Layers Neural Networks
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
• Characterizing Implicit Bias in Terms of Optimization Geometry
• Nonlinear Approximation and (Deep) ReLU Networks
• A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamics
• SGD Learns the Conjugate Kernel Class of the Network
• Generalization Error Bounds of Gradient Descent for Learning Over-parameterized Deep ReLU Networks
• Spurious Valleys in Two-layer Neural Network Optimization Landscapes
• Gradient Descent Quantizes ReLU Network Features
• Efficient Representation of Low-Dimensional Manifolds using Deep Networks
• Global convergence of neuron birth-death dynamics
• How do infinite width bounded norm networks look in function space?
• Representation by Integrating Reproducing Kernels

Cited by in corpus (16)

• On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networks
• Optimization for deep learning: theory and algorithms
• Banach Space Representer Theorems for Neural Networks and Ridge Splines
• Convergence analysis for gradient flows in the training of artificial neural networks with ReLU activation
• Phase diagram for two-layer ReLU neural networks at infinite-width limit
• The Implicit Bias of Depth: How Incremental Learning Drives Generalization
• CoSE: Compositional Stroke Embeddings
• When Does Preconditioning Help or Hurt Generalization?
• Convex Geometry and Duality of Over-parameterized Neural Networks
• On Sparsity in Overparametrised Shallow ReLU Networks
• Implicit Regularization in ReLU Networks with the Square Loss
• How Implicit Regularization of ReLU Neural Networks Characterizes the Learned Function -- Part I: the 1-D Case of Two Layers with Random First Layer
• Universal scaling laws in the gradient descent training of neural networks
• Plateau Phenomenon in Gradient Descent Training of ReLU networks: Explanation, Quantification and Avoidance
• Implicit Sparse Regularization: The Impact of Depth and Early Stopping
• Mean-field Analysis of Piecewise Linear Solutions for Wide ReLU Networks