Stochastic Particle Gradient Descent for Infinite Ensembles

Abstract

The superior performance of ensemble methods with infinite models are well known. Most of these methods are based on optimization problems in infinite-dimensional spaces with some regularization, for instance, boosting methods and convex neural networks use $L^1$-regularization with the non-negative constraint. However, due to the difficulty of handling $L^1$-regularization, these problems require early stopping or a rough approximation to solve it inexactly. In this paper, we propose a new ensemble learning method that performs in a space of probability measures, that is, our method can handle the $L^1$-constraint and the non-negative constraint in a rigorous way. Such an optimization is realized by proposing a general purpose stochastic optimization method for learning probability measures via parameterization using transport maps on base models. As a result of running the method, a transport map to output an infinite ensemble is obtained, which forms a residual-type network. From the perspective of functional gradient methods, we give a convergence rate as fast as that of a stochastic optimization method for finite dimensional nonconvex problems. Moreover, we show an interior optimality property of a local optimality condition used in our analysis.

arXiv comment33 pages, 1 figure

References in corpus (2)

• Provable Bayesian Inference via Particle Mirror Descent
• A note on the uniformity of the constant in the Poincaré inequality

Cited by in corpus (12)

• Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit
• A Mean-field Analysis of Deep ResNet and Beyond: Towards Provable Optimization Via Overparameterization From Depth
• Gradient Descent can Learn Less Over-parameterized Two-layer Neural Networks on Classification Problems
• Optimal Rates for Averaged Stochastic Gradient Descent under Neural Tangent Kernel Regime
• Large-time asymptotics in deep learning
• Global Convergence of Three-layer Neural Networks in the Mean Field Regime
• Functional Gradient Boosting based on Residual Network Perception
• Ridge Regression with Over-Parametrized Two-Layer Networks Converge to Ridgelet Spectrum
• Particle Dual Averaging: Optimization of Mean Field Neural Networks with Global Convergence Rate Analysis
• The sharp, the flat and the shallow: Can weakly interacting agents learn to escape bad minima?
• Fast Approximation and Estimation Bounds of Kernel Quadrature for Infinitely Wide Models
• Understanding Deflation Process in Over-parametrized Tensor Decomposition