Accelerating LLM Inference via Vector Index Based Output Embeddings

Martin Loretz, Sepp Hochreiter

Large output embedding matrices create a significant memory bandwidth bottleneck during autoregressive decoding, especially for compact LLMs with large multilingual vocabularies. W…