565 citations · 2.1k across the 28 of their papers we have counts for
6 papers · 1 filter
When Do Curricula Work?
Xiaoxia Wu, Ethan Dyer, Behnam Neyshabur
Inspired by human learning, researchers have proposed ordering examples during training based on their difficulty. Both curriculum learning, exposing a network to easier examples e…
Asymptotics of Wide Convolutional Neural Networks
Anders Andreassen, Ethan Dyer
Wide neural networks have proven to be a rich class of architectures for both theory and practice. Motivated by the observation that finite width convolutional networks appear to o…
Whitening and second order optimization both make information in the dataset unusable during training, and can reduce or prevent generalization
Neha S. Wadia, Daniel Duckworth, Samuel S. Schoenholz +2
Machine learning is predicated on the concept of generalization: a model achieving low error on a sufficiently large training set should also perform well on novel samples from the…
Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semantics
Vinay V. Ramasesh, Ethan Dyer, Maithra Raghu
A central challenge in developing versatile machine learning systems is catastrophic forgetting: a model trained on tasks in sequence will suffer significant performance drops on e…
The large learning rate phase of deep learning: the catapult mechanism
Aitor Lewkowycz, Yasaman Bahri, Ethan Dyer +2
The choice of initial learning rate can have a profound effect on the performance of deep networks. We present a class of neural networks with solvable training dynamics, and confi…
Affinity and Diversity: Quantifying Mechanisms of Data Augmentation
Raphael Gontijo-Lopes, Sylvia J. Smullin, Ekin D. Cubuk +1
Though data augmentation has become a standard component of deep neural network training, the underlying mechanism behind the effectiveness of these techniques remains poorly under…