Out-of-Vocabulary Sampling Boosts Speculative Decoding

Nadav Timor, Jonathan Mamou, Oren Pereg +2

Speculative decoding relies on fast and accurate drafters. Recent state-of-the-art language models employ larger and larger vocabularies, which significantly slows down drafters. O…