From the 1 of 13 linked papers with an AI index.
4 papers · 1 filter
DualDecoder: Accelerate Long Context LLM Inference by Predictive Prefetch
Zuning Liang, Zhiyi Yao, Qi Chen +6
DualDecoder is a serving system that predicts and prefetches the key‑value cache entries needed for the next token in long‑context LLM inference, reducing GPU memory overhead and b…
Demystifying the Design Space and Best Practices for Heterogeneous LLM Inference and Serving
Zhixin Wang, Zhengbo Wang, Fangcheng Fu +15
Heterogeneous prefill-decode (PD) inference is now in production: prefill on cost-efficient or supply-available accelerators, decode on bandwidth-strong ones, and KV state crossing…
An Efficient, Reliable and Observable Collective Communication Library in Large-scale GPU Training Clusters
Mingjun Zhang, Xiaohe Hu, Menghao Zhang +21
Large-scale LLM training requires collective communication libraries to exchange data among distributed GPUs. As a company dedicated to building and operating large-scale GPU train…
DistFlow: A Fully Distributed RL Framework for Scalable and Efficient LLM Post-Training
Zhixin Wang, Jiaming Xu, Tianyi Zhou +10
Effectively scaling Reinforcement Learning (RL) is crucial for enhancing the reasoning and alignment of Large Language Models. The massive data and complex execution flows inherent…