208 citations · 299 across the 14 of their papers we have counts for
14 papers
PROFIT: A Specialized Optimizer for Deep Fine Tuning
Anirudh S Chakravarthy, Shuai Kyle Zheng, Xin Huang +4
The fine-tuning of pre-trained models has become ubiquitous in generative AI, computer vision, and robotics. Although much attention has been paid to improving the efficiency of fi…
Contractive error feedback for gradient compression
Bingcong Li, Shuai Zheng, Parameswaran Raman +2
On-device memory concerns in distributed deep learning have become severe due to (i) the growth of model size in multi-GPU training, and (ii) the wide adoption of deep neural netwo…
Federated Learning via Consensus Mechanism on Heterogeneous Data: A New Perspective on Convergence
Shu Zheng, Tiandi Ye, Xiang Li +1
Federated learning (FL) on heterogeneous data (non-IID data) has recently received great attention. Most existing methods focus on studying the convergence guarantees for the globa…
Compressed Communication for Distributed Training: Adaptive Methods and System
Yuchen Zhong, Cong Xie, Shuai Zheng +1
Communication overhead severely hinders the scalability of distributed machine learning systems. Recently, there has been a growing interest in using gradient compression to reduce…
LID 2020: The Learning from Imperfect Data Challenge Results
Yunchao Wei, Shuai Zheng, Ming-Ming Cheng +32
Learning from imperfect data becomes an issue in many industrial applications after the research community has made profound progress in supervised learning from perfectly annotate…
CSER: Communication-efficient SGD with Error Reset
Cong Xie, Shuai Zheng, Oluwasanmi Koyejo +3
The scalability of Distributed Stochastic Gradient Descent (SGD) is today limited by communication bottlenecks. We propose a novel SGD variant: Communication-efficient SGD with Err…
Accelerated Large Batch Optimization of BERT Pretraining in 54 minutes
Shuai Zheng, Haibin Lin, Sheng Zha +1
BERT has recently attracted a lot of attention in natural language understanding (NLU) and achieved state-of-the-art results in various NLU tasks. However, its success requires lar…
GluonCV and GluonNLP: Deep Learning in Computer Vision and Natural Language Processing
Jian Guo, He He, Tong He +13
We present GluonCV and GluonNLP, the deep learning toolkits for computer vision and natural language processing based on Apache MXNet (incubating). These toolkits provide state-of-…
Blockwise Adaptivity: Faster Training and Better Generalization in Deep Learning
Shuai Zheng, James T. Kwok
Stochastic methods with coordinate-wise adaptive stepsize (such as RMSprop and Adam) have been widely used in training deep neural networks. Despite their fast convergence, they ca…
Communication-Efficient Distributed Blockwise Momentum SGD with Error-Feedback
Shuai Zheng, Ziyue Huang, James T. Kwok
Communication overhead is a major bottleneck hampering the scalability of distributed machine learning systems. Recently, there has been a surge of interest in using gradient compr…
Lightweight Stochastic Optimization for Minimizing Finite Sums with Infinite Data
Shuai Zheng, James T. Kwok
Variance reduction has been commonly used in stochastic optimization. It relies crucially on the assumption that the data set is finite. However, when the data are imputed with ran…
Stochastic Variance-Reduced ADMM
Shuai Zheng, James T. Kwok
The alternating direction method of multipliers (ADMM) is a powerful optimization solver in machine learning. Recently, stochastic ADMM has been integrated with variance reduction…
Fast Nonsmooth Regularized Risk Minimization with Continuation
Shuai Zheng, Ruiliang Zhang, James T. Kwok
In regularized risk minimization, the associated optimization problem becomes particularly difficult when both the loss and regularizer are nonsmooth. Existing approaches either ha…
Asynchronous Distributed Semi-Stochastic Gradient Optimization
Ruiliang Zhang, Shuai Zheng, James T. Kwok
With the recent proliferation of large-scale learning problems,there have been a lot of interest on distributed machine learning algorithms, particularly those that are based on st…