8 papers
AdaGC: Enhancing LLM Pretraining Stability via Adaptive Gradient Clipping
Guoxia Wang, Shuai Li, Congliang Chen +5
Loss spikes remain a persistent obstacle in large-scale language model pretraining. While previous research has attempted to identify the root cause of loss spikes by investigating…
MaskPro: Linear-Space Probabilistic Learning for Strict (N:M)-Sparsity on LLMs
Yan Sun, Qixin Zhang, Zhiyuan Yu +3
The rapid scaling of large language models~(LLMs) has made inference efficiency a primary bottleneck in the practical deployment. To address this, semi-structured sparsity offers a…
Provable Sparse Inversion and Token Relabel Enhanced One-shot Federated Learning with ViTs
Li Shen, Xiaolei Hao, Qinglun Li +3
One-Shot Federated Learning, where a central server learns a global model in a single communication round, has emerged as a promising paradigm. However, under extremely non-IID set…
SimReg: Achieving Higher Performance in the Pretraining via Embedding Similarity Regularization
Yan Sun, Guoxia Wang, Jinle Zeng +6
Pretraining large language models (LLMs) with next-token prediction has led to remarkable advances, yet the context-dependent nature of token embeddings in such models results in h…
Rethinking the Personalized Relaxed Initialization in the Federated Learning: Consistency and Generalization
Li Shen, Yan Sun, Dacheng Tao
Federated learning (FL) is a distributed paradigm that coordinates massive local clients to collaboratively train a global model via stage-wise local training processes on the hete…
Stability and Generalization of Push-Sum Based Decentralized Optimization over Directed Graphs
Yifei Liang, Yan Sun, Xiaochun Cao +1
Push-Sum-based decentralized learning enables optimization over directed communication networks, where information exchange may be asymmetric. While convergence properties of such…