Mirror Speculative Decoding: Breaking the Serial Barrier in LLM Inference

Nikhil Bhendawade, Kumari Nishu, Arnav Kundu +3

Speculative decoding accelerates LLM inference by using a draft model to look ahead, but gains are capped by the cost of autoregressive draft generation: increasing draft size elev…