1 citations · 1 across the 7 of their papers we have counts for
3 papers · 1 filter
Small-Scale Experiments: Are We There Yet?
Nicholas Lourie, Kyunghyun Cho, Karen Ullrich +1
Scaling laws promised cost-effective experiments; six years later, they have yet to fully deliver. Instead, researchers have found them unreliable at small scales (starting at 4M p…
Always Learning, Always Mixing: Efficient and Simple Data Mixing All The Time
Michael Y. Hu, Apurva Gandhi, Kyunghyun Cho +2
Data mixing decides how to combine different sources or types of data and is a consequential problem throughout language model training. In pretraining, data composition is a key d…
Neural Neural Scaling Laws
Michael Y. Hu, Jane Pan, Ayush Rajesh Jhaveri +2
Neural scaling laws predict how language model performance improves with increased training inputs. While aggregate metrics like validation loss can follow smooth power-law curves,…