167 citations · 257 across the 7 of their papers we have counts for
7 papers · 1 filter
Understanding How Encoder-Decoder Architectures Attend
Kyle Aitken, Vinay V Ramasesh, Yuan Cao +1
Encoder-decoder networks with attention have proven to be a powerful way to solve many sequence-to-sequence tasks. In these networks, attention aligns encoder and decoder states an…
Training Learned Optimizers with Randomly Initialized Learned Optimizers
Luke Metz, C. Daniel Freeman, Niru Maheswaranathan +1
Learned optimizers are increasingly effective, with performance exceeding that of hand designed optimizers such as Adam~\citep{kingma2014adam} on specific tasks \citep{metz2019unde…
Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves
Luke Metz, Niru Maheswaranathan, C. Daniel Freeman +2
Much as replacing hand-designed features with learned functions has revolutionized how we solve perceptual tasks, we believe learned algorithms will transform how we train models.…
Using a thousand optimization tasks to learn hyperparameter search strategies
Luke Metz, Niru Maheswaranathan, Ruoxi Sun +3
We present TaskSet, a dataset of tasks for use in training and evaluating optimizers. TaskSet is unique in its size and diversity, containing over a thousand tasks ranging from ima…
Using learned optimizers to make models robust to input noise
Luke Metz, Niru Maheswaranathan, Jonathon Shlens +2
State-of-the art vision models can achieve superhuman performance on image classification tasks when testing and training data come from the same distribution. However, when models…
Reverse engineering recurrent networks for sentiment classification reveals line attractor dynamics
Niru Maheswaranathan, Alex Williams, Matthew D. Golub +2
Recurrent neural networks (RNNs) are a widely used tool for modeling sequential data, yet they are often treated as inscrutable black boxes. Given a trained recurrent network, we w…