Newton-ADMM: A Distributed GPU-Accelerated Optimizer for Multiclass Classification Problems
arXiv:1807.07132
Abstract
First-order optimization methods, such as stochastic gradient descent (SGD) and its variants, are widely used in machine learning applications due to their simplicity and low per-iteration costs. However, they often require larger numbers of iterations, with associated communication costs in distributed environments. In contrast, Newton-type methods, while having higher per-iteration costs, typically require a significantly smaller number of iterations, which directly translates to reduced communication costs. In this paper, we present a novel distributed optimizer for classification problems, which integrates a GPU-accelerated Newton-type solver with the global consensus formulation of Alternating Direction of Method Multipliers (ADMM). By leveraging the communication efficiency of ADMM, GPU-accelerated inexact-Newton solver, and an effective spectral penalty parameter selection strategy, we show that our proposed method (i) yields better generalization performance on several classification problems; (ii) significantly outperforms state-of-the-art methods in distributed time to solution; and (iii) offers better scaling on large distributed platforms.
References in corpus (15)
- Adam: A Method for Stochastic Optimization
- ADADELTA: An Adaptive Learning Rate Method
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- Searching for Exotic Particles in High-Energy Physics with Deep Learning
- Revisiting Distributed Synchronous SGD
- AIDE: Fast and Communication Efficient Distributed Optimization
- Sub-Sampled Newton Methods I: Globally Convergent Algorithms
- Sub-Sampled Newton Methods II: Local Convergence Rates
- Adaptive ADMM with Spectral Penalty Parameter Selection
- Adaptive Consensus ADMM for Distributed Optimization
- Second-Order Optimization for Non-Convex Machine Learning: An Empirical Study
- DINGO: Distributed Newton-Type Method for Gradient-Norm Optimization
- GIANT: Globally Improved Approximate Newton Method for Distributed Optimization
- A Distributed Second-Order Algorithm You Can Trust
- DynaNewton - Accelerating Newton's Method for Machine Learning