LOST: Low-rank and Sparse Pre-training for Large Language Models

Jiaxi Li, Lu Yin, Li Shen +6

While large language models (LLMs) have achieved remarkable performance across a wide range of tasks, their massive scale incurs prohibitive computational and memory costs for pre-…