H-Spec: Parallel Speculative Decoding Without a Drafter-Side KV Cache

Weifan Jiang, Krishna Teja Chitty-Venkata, Megan Flynn +6

Speculative decoding losslessly accelerates large language model inference by having a lightweight draft model predict future tokens for verification by the target model. Recent bl…