312 citations · 423 across the 4 of their papers we have counts for
5 papers
Defending Compute Thresholds Against Legal Loopholes
Matteo Pistillo, Pablo Villalobos
Existing legal frameworks on AI rely on training compute thresholds as a proxy to identify potentially-dangerous AI models and trigger increased regulatory attention. In the United…
AI capabilities can be significantly improved without expensive retraining
Tom Davidson, Jean-Stanislas Denain, Pablo Villalobos +1
State-of-the-art AI systems can be significantly improved without expensive retraining via "post-training enhancements"-techniques applied after initial training like fine-tuning t…
Will we run out of data? Limits of LLM scaling based on human-generated data
Pablo Villalobos, Anson Ho, Jaime Sevilla +3
We investigate the potential constraints on LLM scaling posed by the availability of public human-generated text data. We forecast the growing demand for training data based on cur…
Machine Learning Model Sizes and the Parameter Gap
Pablo Villalobos, Jaime Sevilla, Tamay Besiroglu +3
We study trends in model size of notable machine learning systems over time using a curated dataset. From 1950 to 2018, model size in language models increased steadily by seven or…
Compute Trends Across Three Eras of Machine Learning
Jaime Sevilla, Lennart Heim, Anson Ho +3
Compute, data, and algorithmic advances are the three fundamental factors that guide the progress of modern Machine Learning (ML). In this paper we study trends in the most readily…