4 citations · 4 across the 10 of their papers we have counts for
10 papers · 1 filter
Communication-Aware Placement and Pruning for Efficient Mixture-of-Experts Inference
Xiao Shi, Yingying Sun, Jiangsu Du +2
As MoE models scale to hundreds of experts, placement and pruning decisions increasingly dictate communication volume, affecting the performance of distributed inference across GPU…
AlignedServe: Orchestrating Prefix-aware Batching to Build a High-throughput and Computing-efficient LLM Serving System
Fengyao Bai, Hongbin Zhang, Zhitao Chen +3
High-throughput inference serving is essential for applications built on large language models (LLMs). Existing serving frameworks reduce request-level and batch-level bubbles thro…
PipeMax: Enhancing Offline LLM Inference on Commodity GPU Servers
Hongbin Zhang, Taosheng Wei, Jiazhi Jiang +3
Offline LLM inference seeks to maximize request processing under fixed budgets, making commodity GPU servers a promising choice. However, prior work typically considers offloading…
TD-Pipe: Temporally-Disaggregated Pipeline Parallelism Architecture for High-Throughput LLM Inference
Hongbin Zhang, Taosheng Wei, Zhenyi Zheng +3
As the model size continuously increases, pipeline parallelism shows great promise in throughput-oriented LLM inference due to its low demand on communications. However, imbalanced…
Ghidorah: Fast LLM Inference on Edge with Speculative Decoding and Hetero-Core Parallelism
Jinhui Wei, Ye Huang, Yuhui Zhou +3
In-situ LLM inference on end-user devices has gained significant interest due to its privacy benefits and reduced dependency on external infrastructure. However, as the decoding pr…
gLLM: Global Balanced Pipeline Parallelism System for Distributed LLM Serving with Token Throttling
Tianyu Guo, Xianwei Zhang, Jiangsu Du +3
Pipeline parallelism has emerged as a predominant approach for deploying large language models (LLMs) across distributed nodes, owing to its lower communication overhead compared t…