7 papers
ConfigSpec: Profiling-Based Configuration Selection for Distributed Edge--Cloud Speculative LLM Serving
Xiangchen Li, Saeid Ghafouri, Jiakun Fan +3
Speculative decoding enables collaborative Large Language Model (LLM) inference across cloud and edge by separating lightweight token drafting from heavyweight verification. While…
WISP: Waste- and Interference-Suppressed Distributed Speculative LLM Serving at the Edge via Dynamic Drafting and SLO-Aware Batching
Xiangchen Li, Jiakun Fan, Qingyuan Wang +7
As Large Language Models (LLMs) become increasingly accessible to end users, an ever-growing number of inference requests are initiated from edge devices and computed on centralize…
Taming the Memory Footprint Crisis: System Design for Production Diffusion LLM Serving
Jiakun Fan, Yanglin Zhang, Xiangchen Li +1
Diffusion Large Language Models (dLLMs) have emerged as a promising alternative to Autoregressive Models (ARMs), utilizing parallel decoding to overcome sequential bottlenecks. How…
APEX: Asynchronous Parallel CPU-GPU Execution for Online LLM Inference on Constrained GPUs
Jiakun Fan, Yanglin Zhang, Xiangchen Li +1
Deploying large language models (LLMs) for online inference is often constrained by limited GPU memory, particularly due to the growing KV cache during auto-regressive decoding. Hy…
Polymorph: Energy-Efficient Multi-Label Classification for Video Streams on Embedded Devices
Saeid Ghafouri, Mohsen Fayyaz, Xiangchen Li +4
Real-time multi-label video classification on embedded devices is constrained by limited compute and energy budgets. Yet, video streams exhibit structural properties such as label…
SLED: A Speculative LLM Decoding Framework for Efficient Edge Serving
Xiangchen Li, Dimitrios Spatharakis, Saeid Ghafouri +5
The growing gap between the increasing complexity of large language models (LLMs) and the limited computational budgets of edge devices poses a key challenge for efficient on-devic…