Global convergence of neuron birth-death dynamics

Abstract

Neural networks with a large number of parameters admit a mean-field description, which has recently served as a theoretical explanation for the favorable training properties of "overparameterized" models. In this regime, gradient descent obeys a deterministic partial differential equation (PDE) that converges to a globally optimal solution for networks with a single hidden layer under appropriate assumptions. In this work, we propose a non-local mass transport dynamics that leads to a modified PDE with the same minimizer. We implement this non-local dynamics as a stochastic neuronal birth-death process and we prove that it accelerates the rate of convergence in the mean-field limit. We subsequently realize this PDE with two classes of numerical schemes that converge to the mean-field equation, each of which can easily be implemented for neural networks with finite numbers of parameters. We illustrate our algorithms with two models to provide intuition for the mechanism through which convergence is accelerated.

References in corpus (4)

• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning
• Information-Geometric Optimization Algorithms: A Unifying Picture via Invariance Principles
• A new optimal transport distance on the space of finite Radon measures

Cited by in corpus (11)

• Maximum Mean Discrepancy Gradient Flow
• Sinkhorn Divergences for Unbalanced Optimal Transport
• Gradient Dynamics of Shallow Univariate ReLU Networks
• The Unbalanced Gromov Wasserstein Distance: Conic Formulation and Relaxation
• Accelerating Langevin Sampling with Birth-death
• A mean-field analysis of two-player zero-sum games
• Entropy-Transport distances between unbalanced metric measure spaces
• Unbalanced Sobolev Descent
• Temporal-difference learning with nonlinear function approximation: lazy training and mean field regimes
• Dual Training of Energy-Based Models with Overparametrized Shallow Neural Networks
• Towards Optimal Randomized Strategies in Adversarial Example Game