Mind the Memory Gap: Unveiling GPU Bottlenecks in Large-Batch LLM Inference

Pol G. Recasens, Ferran Agullo, Yue Zhu +5

Large language models have been widely adopted across different tasks, but their auto-regressive generation nature often leads to inefficient resource utilization during inference.…