Stochastic Dual Coordinate Ascent with Adaptive Probabilities

Abstract

This paper introduces AdaSDCA: an adaptive variant of stochastic dual coordinate ascent (SDCA) for solving the regularized empirical risk minimization problems. Our modification consists in allowing the method adaptively change the probability distribution over the dual variables throughout the iterative process. AdaSDCA achieves provably better complexity bound than SDCA with the best fixed probability distribution, known as importance sampling. However, it is of a theoretical character as it is expensive to implement. We also propose AdaSDCA+: a practical variant which in our experiments outperforms existing non-adaptive methods.

References in corpus (17)

• No More Pesky Learning Rates
• Mini-Batch Semi-Stochastic Gradient Descent in the Proximal Setting
• Communication-Efficient Distributed Dual Coordinate Ascent
• Stochastic Primal-Dual Coordinate Method for Regularized Empirical Risk Minimization
• Mini-Batch Primal and Dual Methods for SVMs
• Proximal Stochastic Dual Coordinate Ascent
• Stochastic Optimization with Importance Sampling
• Distributed Block Coordinate Descent for Minimizing Partially Separable Functions
• Randomized Dual Coordinate Ascent with Arbitrary Sampling
• Smooth minimization of nonsmooth functions with parallel coordinate descent methods
• A Lower Bound for the Optimization of Finite Sums
• An Accelerated Proximal Coordinate Gradient Method and its Application to Regularized Empirical Risk Minimization
• Coordinate Descent with Arbitrary Sampling I: Algorithms and Complexity
• On Optimal Probabilities in Stochastic Coordinate Descent Methods
• A distributed block coordinate descent method for training $l_1$ regularized linear classifiers
• Asynchronous Stochastic Coordinate Descent: Parallelism and Convergence Properties
• Robust Block Coordinate Descent

Cited by in corpus (18)

• Even Faster Accelerated Coordinate Descent Using Non-Uniform Sampling
• Stochastic Distributed Learning with Gradient Quantization and Variance Reduction
• Stochastic Dual Ascent for Solving Linear Systems
• Importance Sampling for Minibatches
• Primal Method for ERM with Flexible Mini-batching Schemes and Non-convex Losses
• Privacy Preserving Randomized Gossip Algorithms
• Block-proximal methods with spatially adapted acceleration
• Don't Jump Through Hoops and Remove Those Loops: SVRG and Katyusha are Better Without the Outer Loop
• Adaptive Sketch-and-Project Methods for Solving Linear Systems
• Sketch and Project: Randomized Iterative Methods for Linear Systems and Inverting Matrices
• Primal-dual block-proximal splitting for a class of non-convex problems
• Randomized Iterative Methods for Linear Systems: Momentum, Inexactness and Gossip
• Minimal Variance Sampling with Provable Guarantees for Fast Training of Graph Neural Networks
• SGD with Coordinate Sampling: Theory and Practice
• Reducing Runtime by Recycling Samples
• Adam with Bandit Sampling for Deep Learning
• ADASS: Adaptive Sample Selection for Training Acceleration
• Optimization for Supervised Machine Learning: Randomized Algorithms for Data and Parameters