4 papers
RServe: Overlapping Encoding and Prefill for Efficient LMM Inference
Tianyu Guo, Tianming Xu, Xianjie Chen +3
Large multimodal models (LMMs) typically employ an encoding module to transform multimodal data inputs into embeddings, which are then fed to language models for further processing…
Accelerating Edge Inference for Distributed MoE Models with Latency-Optimized Expert Placement
Tian Wu, Liming Wang, Zijian Wen +5
The emergence of Mixture-of-Experts (MoE) has transformed the scaling of large language models by enabling vast model capacity through sparse activation. Yet, converting these perf…
EFIM: Efficient Serving of LLMs for Infilling Tasks with Improved KV Cache Reuse
Tianyu Guo, Hande Dong, Yichong Leng +4
Large language models (LLMs) are often used for infilling tasks, which involve predicting or generating missing information in a given text. These tasks typically require multiple…
gLLM: Global Balanced Pipeline Parallelism System for Distributed LLM Serving with Token Throttling
Tianyu Guo, Xianwei Zhang, Jiangsu Du +3
Pipeline parallelism has emerged as a predominant approach for deploying large language models (LLMs) across distributed nodes, owing to its lower communication overhead compared t…