A Multi-Batch L-BFGS Method for Machine Learning

Abstract

The question of how to parallelize the stochastic gradient descent (SGD) method has received much attention in the literature. In this paper, we focus instead on batch methods that use a sizeable fraction of the training set at each iteration to facilitate parallelism, and that employ second-order information. In order to improve the learning process, we follow a multi-batch approach in which the batch changes at each iteration. This can cause difficulties because L-BFGS employs gradient differences to update the Hessian approximations, and when these gradients are computed using different data points the process can be unstable. This paper shows how to perform stable quasi-Newton updating in the multi-batch setting, illustrates the behavior of the algorithm in a distributed computing platform, and studies its convergence properties for both the convex and nonconvex cases.

NIPS 2016. 31 pages, 22 figures

References in corpus (1)

• SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives

Cited by in corpus (5)

• Straggler Mitigation in Distributed Optimization Through Data Encoding
• Stochastic Calibration of Radio Interferometers
• Trust-Region Algorithms for Training Responses: Machine Learning Methods Using Indefinite Hessian Approximations
• HAMSI: A Parallel Incremental Optimization Algorithm Using Quadratic Approximations for Solving Partially Separable Problems
• Enhance Curvature Information by Structured Stochastic Quasi-Newton Methods