Simple Stochastic Gradient Methods for Non-Smooth Non-Convex Regularized Optimization

Abstract

Our work focuses on stochastic gradient methods for optimizing a smooth non-convex loss function with a non-smooth non-convex regularizer. Research on this class of problem is quite limited, and until recently no non-asymptotic convergence results have been reported. We present two simple stochastic gradient algorithms, for finite-sum and general stochastic optimization problems, which have superior convergence complexities compared to the current state-of-the-art. We also compare our algorithms' performance in practice for empirical risk minimization.

References in corpus (6)

• Nearly unbiased variable selection under minimax concave penalty
• Feature Selection with Annealing for Computer Vision and Big Data Learning
• A Simple Proximal Stochastic Gradient Method for Nonsmooth Nonconvex Optimization
• Analysis of nonsmooth stochastic approximation: the differential inclusion approach
• A successive difference-of-convex approximation method for a class of nonconvex nonsmooth optimization problems
• Stochastic Gradient Descent for Stochastic Doubly-Nonconvex Composite Optimization

Cited by in corpus (5)

• A Hybrid Stochastic Optimization Framework for Stochastic Composite Nonconvex Optimization
• Regret minimization in stochastic non-convex learning via a proximal-gradient approach
• Private and Robust Distributed Nonconvex Optimization via Polynomial Approximation
• Stochastic Proximal Methods for Non-Smooth Non-Convex Constrained Sparse Optimization
• Practical Precoding via Asynchronous Stochastic Successive Convex Approximation