90 citations · 96 across the 7 of their papers we have counts for
9 papers
Formalizing Forgetting: Theory of Forgetting for Online Continual Learning
Renjie Cao, Yifei Cheng, Li Shen
Online continual learning (OCL) requires a learner to acquire a sequence of tasks in a single pass over the data, and catastrophic forgetting of earlier tasks remains its central c…
Sparse Layer Sharpness-Aware Minimization for Efficient Fine-Tuning
Yifei Cheng, Xianglin Yang, Guoxia Wang +5
Sharpness-aware minimization (SAM) seeks the minima with a flat loss landscape to improve the generalization performance in machine learning tasks, including fine-tuning. However,…
LightSAM: Parameter-Agnostic Sharpness-Aware Minimization
Yifei Cheng, Li Shen, Hao Sun +3
Sharpness-Aware Minimization (SAM) optimizer enhances the generalization ability of the machine learning model by exploring the flat minima landscape through weight perturbations.…
Adaptive Weighting Push-SUM for Decentralized Optimization with Statistical Diversity
Yiming Zhou, Yifei Cheng, Linli Xu +1
Statistical diversity is a property of data distribution and can hinder the optimization of a decentralized network. However, the theoretical limitations of the Push-SUM protocol r…
Communication-Efficient Distributed Learning with Local Immediate Error Compensation
Yifei Cheng, Li Shen, Linli Xu +6
Gradient compression with error compensation has attracted significant attention with the target of reducing the heavy communication overhead in distributed learning. However, exis…
DropIT: Dropping Intermediate Tensors for Memory-Efficient DNN Training
Joya Chen, Kai Xu, Yuhui Wang +2
A standard hardware bottleneck when training deep neural networks is GPU memory. The bulk of memory is occupied by caching intermediate tensors for gradient computation in the back…
STL-SGD: Speeding Up Local SGD with Stagewise Communication Period
Shuheng Shen, Yifei Cheng, Jingchang Liu +1
Distributed parallel stochastic gradient descent algorithms are workhorses for large scale machine learning tasks. Among them, local stochastic gradient descent (Local SGD) has att…
Variance Reduced Local SGD with Lower Communication Complexity
Xianfeng Liang, Shuheng Shen, Jingchang Liu +3
To accelerate the training of machine learning models, distributed stochastic gradient descent (SGD) and its variants have been widely adopted, which apply multiple workers in para…
Faster Distributed Deep Net Training: Computation and Communication Decoupled Stochastic Gradient Descent
Shuheng Shen, Linli Xu, Jingchang Liu +2
With the increase in the amount of data and the expansion of model scale, distributed parallel training becomes an important and successful technique to address the optimization ch…