2 papers
cs.DC2025
MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelism
Ruidong Zhu, Ziheng Jiang, Chao Jin +17
Mixture-of-Experts (MoE) showcases tremendous potential to scale large language models (LLMs) with enhanced performance and reduced computational complexity. However, its sparsely…
cs.CL2024
Untie the Knots: An Efficient Data Augmentation Strategy for Long-Context Pre-Training in Language Models
Junfeng Tian, Da Zheng, Yang Cheng +3
Large language models (LLM) have prioritized expanding the context window from which models can incorporate more information. However, training models to handle long contexts prese…