4 papers · 1 filter
ELAS: Efficient Pre-Training of Low-Rank Large Language Models via 2:4 Activation Sparsity
Jiaxi Li, Lu Yin, Li Shen +5
Large Language Models (LLMs) have achieved remarkable capabilities, but their immense computational demands during training remain a critical bottleneck for widespread adoption. Lo…
Model Unmerging: Making Your Models Unmergeable for Secure Model Sharing
Zihao Wang, Enneng Yang, Lu Yin +2
Model merging leverages multiple finetuned expert models to construct a multi-task model with low cost, and is gaining increasing attention. However, as a growing number of finetun…
LOST: Low-rank and Sparse Pre-training for Large Language Models
Jiaxi Li, Lu Yin, Li Shen +6
While large language models (LLMs) have achieved remarkable performance across a wide range of tasks, their massive scale incurs prohibitive computational and memory costs for pre-…
GradientStabilizer:Fix the Norm, Not the Gradient
Tianjin Huang, Zhangyang Wang, Haotian Hu +10
Training instability in modern deep learning systems is frequently triggered by rare but extreme gradient-norm spikes, which can induce oversized parameter updates, corrupt optimiz…