5 papers
Internet of Agents: Fundamentals, Applications, and Challenges
Yuntao Wang, Shaolong Guo, Yanghe Pan +6
With the rapid proliferation of large language models and vision-language models, AI agents have evolved from isolated, task-specific systems into autonomous, interactive entities…
Federated Fine-Tuning of Sparsely-Activated Large Language Models on Resource-Constrained Devices
Fahao Chen, Jie Wan, Peng Li +2
Federated fine-tuning of Mixture-of-Experts (MoE)-based large language models (LLMs) is challenging due to their massive computational requirements and the resource constraints of…
Semi-Clairvoyant Scheduling of Speculative Decoding Requests to Minimize LLM Inference Latency
Ruixiao Li, Fahao Chen, Peng Li
Speculative decoding accelerates Large Language Model (LLM) inference by employing a small speculative model (SSM) to generate multiple candidate tokens and verify them using the L…
SPIN: Accelerating Large Language Model Inference with Heterogeneous Speculative Models
Fahao Chen, Peng Li, Tom H. Luan +2
Speculative decoding has been shown as an effective way to accelerate Large Language Model (LLM) inference by using a Small Speculative Model (SSM) to generate candidate tokens in…
Mell: Memory-Efficient Large Language Model Serving via Multi-GPU KV Cache Management
Liu Qianli, Hong Zicong, Chen Fahao +2
Serving large language models (LLMs) for massive users is challenged by the significant memory footprint of the transient state, known as the key-value (KV) cache, which scales wit…