Jakiro: Boosting Speculative Decoding with Decoupled Multi-Head via MoE

Haiduo Huang, Fuwei Yang, Zhenhua Liu +7

Speculative decoding (SD) accelerates large language model inference by using a smaller draft model to predict multiple tokens, which are then verified in parallel by the larger ta…