Algorithmic progress in language models

Anson Ho, Tamay Besiroglu, Ege Erdil +6

We investigate the rate at which algorithms for pre-training language models have improved since the advent of deep learning. Using a dataset of over 200 language model evaluations…