6 papers
SpecFed: Accelerating Federated LLM Inference with Speculative Decoding and Compressed Transmission
Ce Zheng, Xinghan Wang, Jiahong Ning +3
Federated inference enhances LLM performance in edge computing through weighted averaging of distributed model predictions. However, autoregressive LLM inference requires frequent…
Low-Latency Edge LLM Handover via Joint KV Cache Transfer and Token Prefill
Seunghun Lee, Jihong Park, Ce Zheng +1
Edge deployment of large language models (LLMs) can reduce latency for interactive services, but mobility introduces service interruptions when an user equipment (UE) hands over be…
Fast Collaborative Inference via Distributed Speculative Decoding
Ce Zheng, Ke Zhang, Chen Sun +3
Speculative decoding accelerates large language model (LLM) inference by allowing a small draft model to predict multiple future tokens for verification by a larger target model. I…
Communication-Efficient Collaborative LLM Inference via Distributed Speculative Decoding
Ce Zheng, Tingting Yang
Speculative decoding is an emerging technique that accelerates large language model (LLM) inference by allowing a smaller draft model to predict multiple tokens in advance, which a…
DSSD: Efficient Edge-Device LLM Deployment and Collaborative Inference via Distributed Split Speculative Decoding
Jiahong Ning, Ce Zheng, Tingting Yang
Large language models (LLMs) have transformed natural language processing but face critical deployment challenges in device-edge systems due to resource limitations and communicati…
EdgePrompt: A Distributed Key-Value Inference Framework for LLMs in 6G Networks
Jiahong Ning, Pengyan Zhu, Ce Zheng +3
As sixth-generation (6G) networks advance, large language models (LLMs) are increasingly integrated into 6G infrastructure to enhance network management and intelligence. However,…