787 citations · 2.7k across the 64 of their papers we have counts for
4 papers · 2 filters
A Closer Look at Deep Learning Heuristics: Learning rate restarts, Warmup and Distillation
Akhilesh Gotmare, Nitish Shirish Keskar, Caiming Xiong +1
The convergence rate and final performance of common deep learning models have significantly benefited from heuristics such as learning rate schedules, knowledge distillation, skip…
Identifying Generalization Properties in Neural Networks
Huan Wang, Nitish Shirish Keskar, Caiming Xiong +1
While it has not yet been proven, empirical evidence suggests that model generalization is related to local properties of the optima which can be described via the Hessian. We conn…
Augmented Cyclic Adversarial Learning for Low Resource Domain Adaptation
Ehsan Hosseini-Asl, Yingbo Zhou, Caiming Xiong +1
Training a model to perform a task typically requires a large amount of data from the domains in which the task will be applied. However, it is often the case that data are abundan…
Using Mode Connectivity for Loss Landscape Analysis
Akhilesh Gotmare, Nitish Shirish Keskar, Caiming Xiong +1
Mode connectivity is a recently introduced frame- work that empirically establishes the connected- ness of minima by finding a high accuracy curve between two independently trained…