5 papers
SR-OPSD: Self-Referenced On-Policy Self-Distillation
Zhuo Sun, Entong Li, Yanlong Zhao +9
On-policy self-distillation (OPSD) converts feedback into dense token-level supervision on student-generated trajectories, complementing reinforcement learning with sparse outcome…
High-Throughput LLM inference on Heterogeneous Clusters
Yi Xiong, Jinqi Huang, Wenjie Huang +6
Nowadays, many companies possess various types of AI accelerators, forming heterogeneous clusters. Efficiently leveraging these clusters for high-throughput large language model (L…
SLO-Aware Scheduling for Large Language Model Inferences
Jinqi Huang, Yi Xiong, Xuebing Yu +4
Large language models (LLMs) have revolutionized applications such as code completion, chatbots, and online classification. To elevate user experiences, service level objectives (S…
WindVE: Collaborative CPU-NPU Vector Embedding
Jinqi Huang, Xuebing Yu, Yi Xiong +4
Retrieval-Augmented Generation is a technology that enhances large language models by integrating information retrieval. In the industry, inference services based on LLMs are highl…
LocMoE: A Low-Overhead MoE for Large Language Model Training
Jing Li, Zhijie Sun, Xuan He +6
The Mixtures-of-Experts (MoE) model is a widespread distributed and integrated learning method for large language models (LLM), which is favored due to its ability to sparsify and…