Draft on the Fly: Adaptive Self-Speculative Decoding using Cosine Similarity

Michael R. Metel, Peng Lu, Boxing Chen +2

We present a simple on the fly method for faster inference of large language models. Unlike other (self-)speculative decoding techniques, our method does not require fine-tuning or…