CSV-Decode: Certifiable Sub-Vocabulary Decoding for Efficient Large Language Model Inference

Dong Liu, Shu Wang, Yanxuan Yu +2

Large language models face significant computational bottlenecks during inference due to the expensive output layer computation over large vocabularies. We present CSV-Decode, a no…