Ridge Regression with Over-Parametrized Two-Layer Networks Converge to Ridgelet Spectrum

Abstract

Characterization of local minima draws much attention in theoretical studies of deep learning. In this study, we investigate the distribution of parameters in an over-parametrized finite neural network trained by ridge regularized empirical square risk minimization (RERM). We develop a new theory of ridgelet transform, a wavelet-like integral transform that provides a powerful and general framework for the theoretical study of neural networks involving not only the ReLU but general activation functions. We show that the distribution of the parameters converges to a spectrum of the ridgelet transform. This result provides a new insight into the characterization of the local minima of neural networks, and the theoretical background of an inductive bias theory based on lazy regimes. We confirm the visual resemblance between the parameter distribution trained by SGD, and the ridgelet spectrum calculated by numerical integration through numerical experiments with finite models.

arXiv commentpublished at AISTATS2021

References in corpus (15)

• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• A Mean Field View of the Landscape of Two-Layers Neural Networks
• Exponential expressivity in deep neural networks through transient chaos
• The loss surface of deep and wide neural networks
• Characterizing Implicit Bias in Terms of Optimization Geometry
• No bad local minima: Data independent training error guarantees for multilayer neural networks
• The Emergence of Spectral Universality in Deep Networks
• Globally Optimal Gradient Descent for a ConvNet with Gaussian Inputs
• Implicit Regularization in Deep Learning
• Norm-Based Capacity Control in Neural Networks
• Depth Creates No Bad Local Minima
• Identity Matters in Deep Learning
• Maximum Mean Discrepancy Gradient Flow
• Stochastic Particle Gradient Descent for Infinite Ensembles
• The global optimum of shallow neural network is attained by ridgelet transform

Cited by in corpus (1)

• A unified Fourier slice method to derive ridgelet transform for a variety of depth-2 neural networks