125 citations · 346 across the 41 of their papers we have counts for
4 papers · 1 filter
Pareto Frontiers in Neural Feature Learning: Data, Compute, Width, and Luck
Benjamin L. Edelman, Surbhi Goel, Sham Kakade +2
In modern deep learning, algorithmic choices (such as width, depth, and learning rate) are known to modulate nuanced resource tradeoffs. This work investigates how these complexiti…
Corgi^2: A Hybrid Offline-Online Approach To Storage-Aware Data Shuffling For SGD
Etay Livne, Gal Kaplun, Eran Malach +1
When using Stochastic Gradient Descent (SGD) for training machine learning models, it is often crucial to provide the model with examples sampled at random from the dataset. Howeve…
Auto-Regressive Next-Token Predictors are Universal Learners
Eran Malach
Large language models display remarkable capabilities in logical and mathematical reasoning, allowing them to solve complex tasks. Interestingly, these abilities emerge in networks…
Less is More: Selective Layer Finetuning with SubTuning
Gal Kaplun, Andrey Gurevich, Tal Swisa +3
Finetuning a pretrained model has become a standard approach for training neural networks on novel tasks, resulting in fast convergence and improved performance. In this work, we s…