Fast Inference via Hierarchical Speculative Decoding

Clara Mohri, Haim Kaplan, Tal Schuster +2

Transformer language models generate text autoregressively, making inference latency proportional to the number of tokens generated. Speculative decoding reduces this latency witho…