Align Attention Heads Before Merging Them: An Effective Way for Converting MHA to GQA

Qingyun Jin, Xiaohui Song, Feng Zhou +1

Large language models (LLMs) have demonstrated exceptional performance across diverse natural language processing tasks. However, as the model size and the input sequence's length…