RADAR: Accelerating Large Language Model Inference With RL-Based Dynamic Draft Trees

Junjie Ma, Jinlong Li

Inference with modern Large Language Models (LLMs) is expensive and slow, and speculative sampling has emerged as an effective solution to this problem, however, the number of the…