45 citations · 191 across the 17 of their papers we have counts for
6 papers · 1 filter
VeLO: Training Versatile Learned Optimizers by Scaling Up
Luke Metz, James Harrison, C. Daniel Freeman +8
While deep learning models have replaced hand-designed features across many domains, these models are still trained with hand-designed optimizers. In this work, we leverage the sam…
Practical tradeoffs between memory, compute, and performance in learned optimizers
Luke Metz, C. Daniel Freeman, James Harrison +2
Optimization plays a costly and crucial role in developing machine learning systems. In learned optimizers, the few hyperparameters of commonly used hand-designed optimizers, e.g.…
Gradients are Not All You Need
Luke Metz, C. Daniel Freeman, Samuel S. Schoenholz +1
Differentiable programming techniques are widely used in the community and are responsible for the machine learning renaissance of the past several decades. While these methods are…
Training Learned Optimizers with Randomly Initialized Learned Optimizers
Luke Metz, C. Daniel Freeman, Niru Maheswaranathan +1
Learned optimizers are increasingly effective, with performance exceeding that of hand designed optimizers such as Adam~\citep{kingma2014adam} on specific tasks \citep{metz2019unde…
Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves
Luke Metz, Niru Maheswaranathan, C. Daniel Freeman +2
Much as replacing hand-designed features with learned functions has revolutionized how we solve perceptual tasks, we believe learned algorithms will transform how we train models.…
Using a thousand optimization tasks to learn hyperparameter search strategies
Luke Metz, Niru Maheswaranathan, Ruoxi Sun +3
We present TaskSet, a dataset of tasks for use in training and evaluating optimizers. TaskSet is unique in its size and diversity, containing over a thousand tasks ranging from ima…