Rethinking LLM Inference Bottlenecks: Insights from Latent Attention and Mixture-of-Experts

Sungmin Yun, Seonyong Park, Hwayong Nam +10

Computational workloads composing traditional transformer models are starkly bifurcated. Multi-Head Attention (MHA) and Grouped-Query Attention are memory-bound due to low arithmet…