7 papers
Curvature-Conditioned Multiscale Momentum with Sphere Constraints for LLM Pretraining
Shuchen Zhu, Yuxin Fang, Mingze Wang +1
Pretraining accounts for a large fraction of the total computational cost in LLM training. However, noise-dominant gradients and the highly ill-conditioned loss landscape bring sev…
Accelerating LLM Pre-Training through Flat-Direction Dynamics Enhancement
Shuchen Zhu, Rizhen Hu, Mingze Wang +4
Pre-training Large Language Models requires immense computational resources, making optimizer efficiency essential. The optimization landscape is highly anisotropic, with loss redu…
A Memory Efficient Randomized Subspace Optimization Method for Training Large Language Models
Yiming Chen, Yuan Zhang, Yin Liu +2
The memory challenges associated with training Large Language Models (LLMs) have become a critical concern, particularly when using the Adam optimizer. To address this issue, numer…
GWT: Scalable Optimizer State Compression for Large Language Model Training
Ziqing Wen, Ping Luo, Jiahuan Wang +4
Training large language models (LLMs) requires substantial memory, a significant fraction of which is consumed by the moment states maintained by adaptive optimizers such as Adam.…
Enhancing Zeroth-order Fine-tuning for Language Models with Low-rank Structures
Yiming Chen, Yuan Zhang, Liyuan Cao +2
Parameter-efficient fine-tuning (PEFT) significantly reduces memory costs when adapting large language models (LLMs) for downstream applications. However, traditional first-order (…
Sharper Convergence Guarantees for Federated Learning with Partial Model Personalization
Yiming Chen, Liyuan Cao, Kun Yuan +1
Partial model personalization, which encompasses both shared and personal variables in its formulation, is a critical optimization problem in federated learning. It balances indivi…
CLLD: Contrastive Learning with Label Distance for Text Classification
Jinhe Lan, Qingyuan Zhan, Chenhao Jiang +2
Existed pre-trained models have achieved state-of-the-art performance on various text classification tasks. These models have proven to be useful in learning universal language rep…