6 papers
ReMoE: Boosting Expert Reuse through Router Fine-Tuning in Memory-Constrained MoE LLM Inference
Xiongwei Zhu, Xiaojian Liao, Tianyang Jiang +3
Fine-grained Mixture-of-Experts (MoE) models sparsely activate only a subset of experts per token, reducing activated computation while maintaining high model capacity. However, in…
Soft Conflict-Resolution Decision Transformer for Offline Multi-Task Reinforcement Learning
Shudong Wang, Xinfei Wang, Chenhao Zhang +4
Multi-task reinforcement learning (MTRL) seeks to learn a unified policy for diverse tasks, but often suffers from gradient conflicts across tasks. Existing masking-based methods a…
DeepCEE: Efficient Cross-Region Model Distributed Training System under Heterogeneous GPUs and Networks
Jinquan Wang, Xiaojian Liao, Xuzhao Liu +7
Most existing training systems focus on a single region. In contrast, we envision that cross-region training offers more flexible GPU resource allocation and yields significant pot…
CaMDN: Enhancing Cache Efficiency for Multi-tenant DNNs on Integrated NPUs
Tianhao Cai, Liang Wang, Limin Xiao +4
With the rapid development of DNN applications, multi-tenant execution, where multiple DNNs are co-located on a single SoC, is becoming a prevailing trend. Although many methods ar…
CoServe: Efficient Collaboration-of-Experts (CoE) Model Inference with Limited Memory
Jiashun Suo, Xiaojian Liao, Limin Xiao +4
Large language models like GPT-4 are resource-intensive, but recent advancements suggest that smaller, specialized experts can outperform the monolithic models on specific tasks. T…
PipeBoost: Resilient Pipelined Architecture for Fast Serverless LLM Scaling
Chongpeng Liu, Xiaojian Liao, Hancheng Liu +2
This paper presents PipeBoost, a low-latency LLM serving system for multi-GPU (serverless) clusters, which can rapidly launch inference services in response to bursty requests with…