Beyond Tokens: Semantic-Aware Speculative Decoding for Efficient Inference by Probing Internal States

Ximing Dong, Shaowei Wang, Dayi Lin +2

Large Language Models (LLMs) achieve strong performance across many tasks but suffer from high inference latency due to autoregressive decoding. The issue is exacerbated in Large R…