Automatic Task Detection and Heterogeneous LLM Speculative Decoding

Danying Ge, Jianhua Gao, Qizhi Jiang +2

Speculative decoding, which combines a draft model with a target model, has emerged as an effective approach to accelerate large language model (LLM) inference. However, existing m…