3 papers
cs.LG2025
Hyperparameter Loss Surfaces Are Simple Near their Optima
Nicholas Lourie, He He, Kyunghyun Cho
Hyperparameters greatly impact models' capabilities; however, modern models are too large for extensive search. Instead, researchers design recipes that train well across scales ba…
cs.CL2025
Scaling Laws Are Unreliable for Downstream Tasks: A Reality Check
Nicholas Lourie, Michael Y. Hu, Kyunghyun Cho
Downstream scaling laws aim to predict task performance at larger scales from the model's performance at smaller scales. Whether such prediction should be possible is unclear: some…
cs.LG2024
Aioli: A Unified Optimization Framework for Language Model Data Mixing
Mayee F. Chen, Michael Y. Hu, Nicholas Lourie +2
Language model performance depends on identifying the optimal mixture of data groups to train on (e.g., law, code, math). Prior work has proposed a diverse set of methods to effici…