Particle Dual Averaging: Optimization of Mean Field Neural Networks with Global Convergence Rate Analysis
arXiv:2012.15477
Abstract
We propose the particle dual averaging (PDA) method, which generalizes the dual averaging method in convex optimization to the optimization over probability distributions with quantitative runtime guarantee. The algorithm consists of an inner loop and outer loop: the inner loop utilizes the Langevin algorithm to approximately solve for a stationary distribution, which is then optimized in the outer loop. The method can thus be interpreted as an extension of the Langevin algorithm to naturally handle nonlinear functional on the probability space. An important application of the proposed method is the optimization of neural network in the mean field regime, which is theoretically attractive due to the presence of nonlinear feature learning, but quantitative convergence rate can be challenging to obtain. By adapting finite-dimensional convex optimization theory into the space of measures, we analyze PDA in regularized empirical / expected risk minimization, and establish quantitative global convergence in learning two-layer mean field neural networks under more general settings. Our theoretical results are supported by numerical simulations on neural networks with reasonable size.
arXiv commentNeurIPS 2021
References in corpus (30)
- Scikit-learn: Machine Learning in Python
- Estimating Mutual Information
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- A Convergence Theory for Deep Learning via Over-Parameterization
- A Mean Field View of the Landscape of Two-Layers Neural Networks
- Gradient Descent Provably Optimizes Over-parameterized Neural Networks
- Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks
- Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysis
- Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionality
- What Can ResNet Learn Efficiently, Going Beyond Kernels?
- Non-asymptotic convergence analysis for the Unadjusted Langevin Algorithm
- Convergence of Langevin MCMC in KL-divergence
- Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networks
- Provable Bayesian Inference via Particle Mirror Descent
- A mean-field limit for certain deep neural networks
- Ensemble Kalman Sampler: mean-field limit and convergence analysis
- Rapid Convergence of the Unadjusted Langevin Algorithm: Isoperimetry Suffices
- Mean-Field Langevin Dynamics and Energy Landscape of Neural Networks
- A Generalized Neural Tangent Kernel Analysis for Two-layer Neural Networks
- Stochastic Particle Gradient Descent for Infinite Ensembles
- Stochastic Runge-Kutta Accelerates Langevin Monte Carlo and Beyond
- A Mean-field Analysis of Deep ResNet and Beyond: Towards Provable Optimization Via Overparameterization From Depth
- Deep learning is adaptive to intrinsic dimensionality of model smoothness in anisotropic Besov space
- Mixing Time Guarantees for Unadjusted Hamiltonian Monte Carlo
- Convergence of unadjusted Hamiltonian Monte Carlo for mean-field models
- On the Convergence of Langevin Monte Carlo: The Interplay between Tail Growth and Smoothness
- Gradient Descent can Learn Less Over-parameterized Two-layer Neural Networks on Classification Problems
- Learning Parities with Neural Networks
- Learning Over-Parametrized Two-Layer ReLU Neural Networks beyond NTK
- Global Convergence of Three-layer Neural Networks in the Mean Field Regime