Challenges and Research Directions for Large Language Model Inference Hardware

Xiaoyu Ma, David Patterson

Large Language Model (LLM) inference is hard. The autoregressive Decode phase of the underlying Transformer model makes LLM inference fundamentally different from training. Exacerb…