17 citations · 54 across the 14 of their papers we have counts for
9 papers · 1 filter
Stochastic Normalized Gradient Descent with Momentum for Large-Batch Training
Shen-Yi Zhao, Chang-Wei Shi, Yin-Peng Xie +1
Stochastic gradient descent~(SGD) and its variants have been the dominating optimization methods in machine learning. Compared to SGD with small-batch training, SGD with large-batc…
Stagewise Enlargement of Batch Size for SGD-based Learning
Shen-Yi Zhao, Yin-Peng Xie, Wu-Jun Li
Existing research shows that the batch size can seriously affect the performance of stochastic gradient descent~(SGD) based learning, including training speed and generalization ab…
ADASS: Adaptive Sample Selection for Training Acceleration
Shen-Yi Zhao, Hao Gao, Wu-Jun Li
Stochastic gradient decent~(SGD) and its variants, including some accelerated variants, have become popular for training in machine learning. However, in all existing SGD and its v…
On the Convergence of Memory-Based Distributed SGD
Shen-Yi Zhao, Hao Gao, Wu-Jun Li
Distributed stochastic gradient descent~(DSGD) has been widely used for optimizing large-scale machine learning models, including both convex and non-convex models. With the rapid…
Global Momentum Compression for Sparse Communication in Distributed Learning
Chang-Wei Shi, Shen-Yi Zhao, Yin-Peng Xie +2
With the rapid growth of data, distributed momentum stochastic gradient descent~(DMSGD) has been widely used in distributed learning, especially for training large-scale deep model…
Proximal SCOPE for Distributed Sparse Learning: Better Data Partition Implies Faster Convergence Rate
Shen-Yi Zhao, Gong-Duo Zhang, Ming-Wei Li +1
Distributed sparse learning with a cluster of multiple machines has attracted much attention in machine learning, especially for large-scale applications with high-dimensional data…
Lock-Free Optimization for Non-Convex Problems
Shen-Yi Zhao, Gong-Duo Zhang, Wu-Jun Li
Stochastic gradient descent~(SGD) and its variants have attracted much attention in machine learning due to their efficiency and effectiveness for optimization. To handle large-sca…
SCOPE: Scalable Composite Optimization for Learning on Spark
Shen-Yi Zhao, Ru Xiang, Ying-Hao Shi +2
Many machine learning models, such as logistic regression~(LR) and support vector machine~(SVM), can be formulated as composite optimization problems. Recently, many distributed st…
Fast Asynchronous Parallel Stochastic Gradient Decent
Shen-Yi Zhao, Wu-Jun Li
Stochastic gradient descent~(SGD) and its variants have become more and more popular in machine learning due to their efficiency and effectiveness. To handle large-scale problems,…