Speculative Sampling with Reinforcement Learning

Chenan Wang, Daniel H. Shi, Haipeng Chen

Inference time latency has remained an open challenge for real world applications of large language models (LLMs). State-of-the-art (SOTA) speculative sampling (SpS) methods for LL…