103 citations · 139 across the 9 of their papers we have counts for
9 papers
Benchmarking Neural Network Training Algorithms
George E. Dahl, Frank Schneider, Zachary Nado +22
Training algorithms, broadly construed, are an essential part of every deep learning pipeline. Training algorithm improvements that speed up training across a wide variety of workl…
Adaptive Gradient Methods at the Edge of Stability
Jeremy M. Cohen, Behrooz Ghorbani, Shankar Krishnan +8
Very little is known about the training dynamics of adaptive gradient methods like Adam in deep learning. In this paper, we shed light on the behavior of these algorithms in the fu…
A Unifying View on Implicit Bias in Training Linear Neural Networks
Chulhee Yun, Shankar Krishnan, Hossein Mobahi
We study the implicit bias of gradient flow (i.e., gradient descent with infinitesimal step size) on linear neural network training. We propose a tensor formulation of neural netwo…
Weak and Strong Gradient Directions: Explaining Memorization, Generalization, and Hardness of Examples at Scale
Piotr Zielinski, Shankar Krishnan, Satrajit Chatterjee
Coherent Gradients (CGH) is a recently proposed hypothesis to explain why over-parameterized neural networks trained with gradient descent generalize well even though they have suf…
Filter Response Normalization Layer: Eliminating Batch Dependence in the Training of Deep Neural Networks
Saurabh Singh, Shankar Krishnan
Batch Normalization (BN) uses mini-batch statistics to normalize the activations during training, introducing dependence between mini-batch elements. This dependency can hurt the p…
An Investigation into Neural Net Optimization via Hessian Eigenvalue Density
Behrooz Ghorbani, Shankar Krishnan, Ying Xiao
To understand the dynamics of optimization in deep neural networks, we develop a tool to study the evolution of the entire Hessian spectrum throughout the optimization process. Usi…