4 papers
SR-OPSD: Self-Referenced On-Policy Self-Distillation
Zhuo Sun, Entong Li, Yanlong Zhao +7
On-policy self-distillation (OPSD) converts feedback into dense token-level supervision on trajectories generated by the policy to be optimized, providing a useful complement to re…
WindVE: Collaborative CPU-NPU Vector Embedding
Jinqi Huang, Xuebing Yu, Yi Xiong +4
Retrieval-Augmented Generation is a technology that enhances large language models by integrating information retrieval. In the industry, inference services based on LLMs are highl…
SLO-Aware Scheduling for Large Language Model Inferences
Jinqi Huang, Yi Xiong, Xuebing Yu +4
Large language models (LLMs) have revolutionized applications such as code completion, chatbots, and online classification. To elevate user experiences, service level objectives (S…
High-Throughput LLM inference on Heterogeneous Clusters
Yi Xiong, Jinqi Huang, Wenjie Huang +6
Nowadays, many companies possess various types of AI accelerators, forming heterogeneous clusters. Efficiently leveraging these clusters for high-throughput large language model (L…