789 citations · 3.9k across the 31 of their papers we have counts for
5 papers · 1 filter
A Closer Look at Deep Learning Heuristics: Learning rate restarts, Warmup and Distillation
Akhilesh Gotmare, Nitish Shirish Keskar, Caiming Xiong +1
The convergence rate and final performance of common deep learning models have significantly benefited from heuristics such as learning rate schedules, knowledge distillation, skip…
Identifying Generalization Properties in Neural Networks
Huan Wang, Nitish Shirish Keskar, Caiming Xiong +1
While it has not yet been proven, empirical evidence suggests that model generalization is related to local properties of the optima which can be described via the Hessian. We conn…
The Natural Language Decathlon: Multitask Learning as Question Answering
Bryan McCann, Nitish Shirish Keskar, Caiming Xiong +1
Deep learning has improved performance on many natural language processing (NLP) tasks individually. However, general NLP models cannot emerge within a paradigm that focuses on the…
Using Mode Connectivity for Loss Landscape Analysis
Akhilesh Gotmare, Nitish Shirish Keskar, Caiming Xiong +1
Mode connectivity is a recently introduced frame- work that empirically establishes the connected- ness of minima by finding a high accuracy curve between two independently trained…
An Analysis of Neural Language Modeling at Multiple Scales
Stephen Merity, Nitish Shirish Keskar, Richard Socher
Many of the leading approaches in language modeling introduce novel, complex and specialized architectures. We take existing state-of-the-art word level language models based on LS…