Speculative Sampling via Exponential Races

Szymon Kobus, Deniz Gündüz

Speculative decoding accelerates large language model inference using a smaller draft model. In this paper, we establish a surprising connection between speculative decoding and ch…