Utility-Driven Speculative Decoding for Mixture-of-Experts

Anish Saxena, Po-An Tsai, Hritvik Taneja +2

GPU memory bandwidth is the main bottleneck for low-latency Large Language Model (LLM) inference. Speculative decoding leverages idle GPU compute by using a lightweight drafter to…