399 citations · 3.4k across the 95 of their papers we have counts for
12 papers · 1 filter
Inexact Proximal Cubic Regularized Newton Methods for Convex Optimization
Chaobing Song, Ji Liu, Yong Jiang
In this paper, we use Proximal Cubic regularized Newton Methods (PCNM) to optimize the sum of a smooth convex function and a non-smooth convex function, where we use inexact gradie…
Stochastic Primal-Dual Method for Empirical Risk Minimization with Per-Iteration Complexity
Conghui Tan, Tong Zhang, Shiqian Ma +1
Regularized empirical risk minimization problem with linear predictor appears frequently in machine learning. In this paper, we propose a new stochastic primal-dual method to solve…
Revisit Batch Normalization: New Understanding from an Optimization View and a Refinement via Composition Optimization
Xiangru Lian, Ji Liu
Batch Normalization (BN) has been used extensively in deep learning to achieve faster training process and better resulting models. However, whether BN works strongly depends on ho…
Stochastically Controlled Stochastic Gradient for the Convex and Non-convex Composition problem
Liu Liu, Ji Liu, Cho-Jui Hsieh +1
In this paper, we consider the convex and non-convex composition problem with the structure , where $G( x )=\frac{1}{n}\sum\…
Asynchronous Decentralized Parallel Stochastic Gradient Descent
Xiangru Lian, Wei Zhang, Ce Zhang +1
Most commonly used distributed machine learning systems are either synchronous or centralized asynchronous. Synchronous algorithms like AllReduce-SGD perform poorly in a heterogene…
Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent
Xiangru Lian, Ce Zhang, Huan Zhang +3
Most distributed machine learning systems nowadays, including TensorFlow and CNTK, are built in a centralized fashion. One bottleneck of centralized algorithms lies on high communi…
Finite-sum Composition Optimization via Variance Reduced Gradient Descent
Xiangru Lian, Mengdi Wang, Ji Liu
The stochastic composition optimization proposed recently by Wang et al. [2014] minimizes the objective with the compositional expectation form: $\min_x~(\mathbb{E}_iF_i \circ \mat…
Accelerating Stochastic Composition Optimization
Mengdi Wang, Ji Liu, Ethan X. Fang
Consider the stochastic composition optimization problem where the objective is a composition of two expected-value functions. We propose a new stochastic first-order method, namel…
A Comprehensive Linear Speedup Analysis for Asynchronous Stochastic Parallel Optimization from Zeroth-Order to First-Order
Xiangru Lian, Huan Zhang, Cho-Jui Hsieh +2
Asynchronous parallel optimization received substantial successes and extensive attention recently. One of core theoretical questions is how much speedup (or benefit) the asynchron…
Asynchronous Parallel Stochastic Gradient for Nonconvex Optimization
Xiangru Lian, Yijun Huang, Yuncheng Li +1
Asynchronous parallel implementations of stochastic gradient (SG) have been broadly used in solving deep neural network and received many successes in practice recently. However, e…
Asynchronous Stochastic Coordinate Descent: Parallelism and Convergence Properties
Ji Liu, Stephen J. Wright
We describe an asynchronous parallel stochastic proximal coordinate descent algorithm for minimizing a composite objective function, which consists of a smooth convex function plus…
An Asynchronous Parallel Stochastic Coordinate Descent Algorithm
Ji Liu, Stephen J. Wright, Christopher Ré +2
We describe an asynchronous parallel stochastic coordinate descent algorithm for minimizing smooth unconstrained or separably constrained functions. The method achieves a linear co…