Accelerating Production LLMs with Combined Token/Embedding Speculators

Davis Wertheimer, Joshua Rosenkranz, Thomas Parnell +4

This technical report describes the design and training of novel speculative decoding draft models, for accelerating the inference speeds of large language models in a production e…