GQA-μP: The maximal parameterization update for grouped query attention

Kyle R. Chickering, Huijuan Wang, Mengxi Wu +7

Hyperparameter transfer across model architectures dramatically reduces the amount of compute necessary for tuning large language models (LLMs). The maximal update parameterization…