29 citations · 47 across the 3 of their papers we have counts for
3 papers
Algorithmic progress in language models
Anson Ho, Tamay Besiroglu, Ege Erdil +6
We investigate the rate at which algorithms for pre-training language models have improved since the advent of deep learning. Using a dataset of over 200 language model evaluations…
The Compute Divide in Machine Learning: A Threat to Academic Contribution and Scrutiny?
Tamay Besiroglu, Sage Andrus Bergerson, Amelia Michael +3
There are pronounced differences in the extent to which industrial and academic AI labs use computing resources. We provide a data-driven survey of the role of the compute divide i…
Machine Learning Model Sizes and the Parameter Gap
Pablo Villalobos, Jaime Sevilla, Tamay Besiroglu +3
We study trends in model size of notable machine learning systems over time using a curated dataset. From 1950 to 2018, model size in language models increased steadily by seven or…