Speculative Decoding Meets Quantization: Compatibility Evaluation and Hierarchical Framework Design

Yudi Zhang, Weilin Zhao, Xu Han +4

Speculative decoding and quantization effectively accelerate memory-bound inference of large language models. Speculative decoding mitigates the memory bandwidth bottleneck by veri…