SpecEE: Accelerating Large Language Model Inference with Speculative Early Exiting

Jiaming Xu, Jiayi Pan, Yongkang Zhou +5

Early exiting has recently emerged as a promising technique for accelerating large language models (LLMs) by effectively reducing the hardware computation and memory access. In thi…