6 papers
AReaL-Hex: Accommodating Asynchronous RL Training over Heterogeneous GPUs
Ran Yan, Youhe Jiang, Tianyuan Wu +7
Maximizing training throughput and cost-efficiency of RL for LLMs is essential to democratize this advanced technique. One promising but challenging approach is to deploy such a co…
Parallax: Efficient LLM Inference Service over Decentralized Environment
Chris Tong, Youhe Jiang, Gufeng Chen +6
Deploying a large language model (LLM) inference service remains costly because centralized serving depends on specialized GPU clusters and high-bandwidth interconnects in datacent…
Efficient Pre-Training of LLMs via Topology-Aware Communication Alignment on More Than 9600 GPUs
Guoliang He, Youhe Jiang, Wencong Xiao +8
The scaling law for large language models (LLMs) depicts that the path towards machine intelligence necessitates training at large scale. Thus, companies continuously build large-s…
Cascadia: An Efficient Cascade Serving System for Large Language Models
Youhe Jiang, Fangcheng Fu, Wanru Zhao +4
Recent advances in large language models (LLMs) have intensified the need to deliver both rapid responses and high-quality outputs. More powerful models yield better results but in…
HexGen-2: Disaggregated Generative Inference of LLMs in Heterogeneous Environment
Youhe Jiang, Ran Yan, Binhang Yuan
Disaggregating the prefill and decoding phases represents an effective new paradigm for generative inference of large language models (LLM), which eliminates prefill-decoding inter…
Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs
Youhe Jiang, Fangcheng Fu, Xiaozhe Yao +6
Recent advancements in Large Language Models (LLMs) have led to increasingly diverse requests, accompanied with varying resource (compute and memory) demands to serve them. However…