4 papers
Deadline-Aware Adaptive Prefill Chunking for Efficient Large Language Model Serving
Siyu Song, Qi Bai, Jinbo Hao +3
Continuous batching improves large language model (LLM) serving throughput, but long prompt prefills can delay decode iterations and violate inter-token latency objectives. Chunked…
Automatic Operator-level Parallelism Planning for Distributed Deep Learning -- A Mixed-Integer Programming Approach
Ruifeng She, Bowen Pang, Kai Li +2
As the artificial intelligence community advances into the era of large models with billions of parameters, distributed training and inference have become essential. While various…
Optimizing LLM Inference Throughput via Memory-aware and SLA-constrained Dynamic Batching
Bowen Pang, Kai Li, Feifan Wang
The increasing adoption of large language models (LLMs) necessitates inference serving systems that can deliver both high throughput and low latency. Deploying LLMs with hundreds o…
Hybrid Offline-online Scheduling Method for Large Language Model Inference Optimization
Bowen Pang, Kai Li, Ruifeng She +1
With the development of large language models (LLMs), it has become increasingly important to optimize hardware usage and improve throughput. In this paper, we study the inference…