Optimal Embedding Learning Rate in LLMs: The Effect of Vocabulary Size

Soufiane Hayou, Liyuan Liu

Pretraining large language models is a costly process. To make this process more efficient, several methods have been proposed to optimize model architecture/parametrization and ha…