HeteroSpec: Leveraging Contextual Heterogeneity for Efficient Speculative Decoding

Siran Liu, Yang Ye, Qianchao Zhu +2

Autoregressive decoding inherently limits the inference throughput of Large Language Model (LLM) due to its sequential dependency. Speculative decoding mitigates this by verifying…