24 papers
Gleam: Adaptive Network-Efficient CUDA API Remoting for Cross-Device GPU Sharing over LANs
Zhihao Xu, Hao Zhong, Zeting Zhou +9
This paper aims to enable computation- and communication-efficient GPU sharing across devices within local area networks (LANs), facilitating ubiquitous AI inference on heterogeneo…
CKV: Compressed and Composable KV Cache Reuse for Efficient LLM Inference
Chuheng Du, Junyi Chen, Hanlin Tang +7
Long-context inference is central to modern large language model (LLM) applications such as retrieval-augmented generation and multi-document reasoning. To mitigate the growing inf…
SmartThinker: Progressive Chain-of-Thought Length Calibration for Efficient Large Language Model Reasoning
Chenzhi Hu, Qinzhe Hu, Yuhang Xu +6
Large reasoning models (LRMs) like OpenAI o1 and DeepSeek-R1 achieve high accuracy on complex tasks by adopting long chain-of-thought (CoT) reasoning paths. However, the inherent v…
BubbleSpec: Turning Long-Tail Bubbles into Speculative Rollout Drafts for Synchronous Reinforcement Learning
Yuhang Xu, Kaibin Tian, Yang Tian +6
Reinforcement Learning (RL) has become a cornerstone for improving the performance of Large Language Models (LLMs). However, its rollout phase constitutes a significant efficiency…
Learning to Seek Help: Dynamic Collaboration Between Small and Large Language Models
Hang Zeng, Xiangyu Liu, Yong Hu +5
Large language models (LLMs) offer strong capabilities but raise cost and privacy concerns, whereas small language models (SLMs) facilitate efficient and private local inference ye…
From Myopic Selection to Long-Horizon Awareness: Sequential LLM Routing for Multi-Turn Dialogue
Jiarui Zhang, Xiangyu Liu, Yong Hu +5
Multi-turn dialogue is the predominant form of interaction with large language models (LLMs). While LLM routing is effective in single-turn settings, existing methods fail to maxim…