4 papers
DART: Diffusion-Inspired Speculative Decoding for Fast LLM Inference
Fuliang Liu, Xue Li, Ketai Zhao +7
Speculative decoding is an effective and lossless approach for accelerating LLM inference. However, existing widely adopted model-based draft designs, such as EAGLE3, improve accur…
Accelerating Mixture-of-Experts Inference by Hiding Offloading Latency with Speculative Decoding
Zhibin Wang, Zhonghui Zhang, Yuhang Zhou +8
Recent advancements in Mixture of Experts (MoE) models have significantly increased their parameter scale as well as model performance. Extensive offloading techniques have been pr…
Chordless Structure: A Pathway to Simple and Expressive GNNs
Hongxu Pan, Shuxian Hu, Mo Zhou +5
Researchers have proposed various methods of incorporating more structured information into the design of Graph Neural Networks (GNNs) to enhance their expressiveness. However, the…
Echo: Efficient Co-Scheduling of Hybrid Online-Offline Tasks for Large Language Model Serving
Zhibin Wang, Shipeng Li, Xue Li +7
Large language models have been widely deployed in various applications, encompassing both interactive online tasks and batched offline tasks. Given the burstiness and latency sens…