SD${}^2$: Self-Distilled Sparse Drafters

Mike Lasby, Nish Sinnadurai, Valavan Manohararajah +3

Speculative decoding is a powerful technique for reducing the latency of Large Language Models (LLMs), offering a fault-tolerant framework that enables the use of highly compressed…