29 citations · 42 across the 3 of their papers we have counts for
3 papers
cs.CL2024★ 13 cited
Algorithmic progress in language models
Anson Ho, Tamay Besiroglu, Ege Erdil +6
We investigate the rate at which algorithms for pre-training language models have improved since the advent of deep learning. Using a dataset of over 200 language model evaluations…
cs.LG2023
Power Law Trends in Speedrunning and Machine Learning
Ege Erdil, Jaime Sevilla
We find that improvements in speedrunning world records follow a power law pattern. Using this observation, we answer an outstanding question from previous work: How do we improve…
cs.LG2022★ 29 cited
Machine Learning Model Sizes and the Parameter Gap
Pablo Villalobos, Jaime Sevilla, Tamay Besiroglu +3
We study trends in model size of notable machine learning systems over time using a curated dataset. From 1950 to 2018, model size in language models increased steadily by seven or…