Distributed Inexact Damped Newton Method: Data Partitioning and Load-Balancing
arXiv:1603.05191
Abstract
In this paper we study inexact dumped Newton method implemented in a distributed environment. We start with an original DiSCO algorithm [Communication-Efficient Distributed Optimization of Self-Concordant Empirical Loss, Yuchen Zhang and Lin Xiao, 2015]. We will show that this algorithm may not scale well and propose an algorithmic modifications which will lead to less communications, better load-balancing and more efficient computation. We perform numerical experiments with an regularized empirical loss minimization instance described by a 273GB dataset.
References in corpus (12)
- HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
- SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives
- A Stochastic Gradient Method with an Exponential Convergence Rate for Finite Training Sets
- Optimal Distributed Online Prediction using Mini-Batches
- Communication Efficient Distributed Optimization using an Approximate Newton-type Method
- Mini-Batch Semi-Stochastic Gradient Descent in the Proximal Setting
- Parallel Coordinate Descent for L1-Regularized Loss Minimization
- Distributed Coordinate Descent Method for Learning with Big Data
- An Asynchronous Parallel Stochastic Coordinate Descent Algorithm
- Distributed Mini-Batch SDCA
- Analysis of Distributed Stochastic Dual Coordinate Ascent
- Distributed Inexact Damped Newton Method: Data Partitioning and Load-Balancing
Cited by in corpus (4)
- Stochastic, Distributed and Federated Optimization for Machine Learning
- Efficient Distributed Hessian Free Algorithm for Large-scale Empirical Risk Minimization via Accumulating Sample Strategy
- Distributed Inexact Damped Newton Method: Data Partitioning and Load-Balancing
- Newton Method over Networks is Fast up to the Statistical Precision