SGLB: Stochastic Gradient Langevin Boosting

Abstract

This paper introduces Stochastic Gradient Langevin Boosting (SGLB) - a powerful and efficient machine learning framework that may deal with a wide range of loss functions and has provable generalization guarantees. The method is based on a special form of the Langevin diffusion equation specifically designed for gradient boosting. This allows us to theoretically guarantee the global convergence even for multimodal loss functions, while standard gradient boosting algorithms can guarantee only local optimum. We also empirically show that SGLB outperforms classic gradient boosting when applied to classification tasks with 0-1 loss function, which is known to be multimodal.

References in corpus (7)

• Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysis
• Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programming
• Global Convergence of Stochastic Gradient Hamiltonian Monte Carlo for Non-Convex Stochastic Optimization: Non-Asymptotic Performance Bounds and Momentum-Based Acceleration
• Efficient Regret Minimization in Non-Convex Games
• Boulevard: Regularized Stochastic Gradient Boosted Trees and Their Limiting Distribution
• On the role of Convexity in Isoperimetry, Spectral-Gap and Concentration
• StochasticRank: Global Optimization of Scale-Free Discrete Functions

Cited by in corpus (2)

• Uncertainty in Gradient Boosting via Ensembles
• StochasticRank: Global Optimization of Scale-Free Discrete Functions