4 papers
Threshold-Based Exclusive Batching for LLM Inference
Weifang Zhang, Yuzhou Nie, Bowen Pang +2
Mixed batching (MB)--interleaving prefill and decode in a single batch--has become the standard scheduling strategy for large language model (LLM) inference due to its efficiency i…
Automatic Operator-level Parallelism Planning for Distributed Deep Learning -- A Mixed-Integer Programming Approach
Ruifeng She, Bowen Pang, Kai Li +2
As the artificial intelligence community advances into the era of large models with billions of parameters, distributed training and inference have become essential. While various…
Optimizing LLM Inference Throughput via Memory-aware and SLA-constrained Dynamic Batching
Bowen Pang, Kai Li, Feifan Wang
The increasing adoption of large language models (LLMs) necessitates inference serving systems that can deliver both high throughput and low latency. Deploying LLMs with hundreds o…
Hybrid Offline-online Scheduling Method for Large Language Model Inference Optimization
Bowen Pang, Kai Li, Ruifeng She +1
With the development of large language models (LLMs), it has become increasingly important to optimize hardware usage and improve throughput. In this paper, we study the inference…