7 papers
Scalable Training of Mixture-of-Experts Models with Megatron Core
Zijie Yan, Hongxiao Bai, Xin Yao +42
Scaling Mixture-of-Experts (MoE) training introduces systems challenges absent in dense models. Because each token activates only a subset of experts, this sparsity allows total pa…
Adacc: An Adaptive Framework Unifying Compression and Activation Recomputation for LLM Training
Ping Chen, Zhuohong Deng, Ping Li +6
Training large language models (LLMs) is often constrained by GPU memory limitations. To alleviate memory pressure, activation recomputation and data compression have been proposed…
CaliDrop: KV Cache Compression with Calibration
Yi Su, Quantong Qiu, Yuechi Zhou +6
Large Language Models (LLMs) require substantial computational resources during generation. While the Key-Value (KV) cache significantly accelerates this process by storing attenti…
Beware of Calibration Data for Pruning Large Language Models
Yixin Ji, Yang Xiang, Juntao Li +5
As large language models (LLMs) are widely applied across various fields, model compression has become increasingly crucial for reducing costs and improving inference efficiency. P…
Beyond Templates: Dynamic Adaptation of Reasoning Demonstrations via Feasibility-Aware Exploration
Yong Wu, Weihang Pan, Ke Li +3
Large language models (LLMs) have shown remarkable reasoning capabilities, yet aligning such abilities to small language models (SLMs) remains a challenge due to distributional mis…
Accurate KV Cache Quantization with Outlier Tokens Tracing
Yi Su, Yuechi Zhou, Quantong Qiu +6
The impressive capabilities of Large Language Models (LLMs) come at the cost of substantial computational resources during deployment. While KV Cache can significantly reduce recom…