425 citations · 1.1k across the 32 of their papers we have counts for
32 papers
Layer-wise Adaptive Step-Sizes for Stochastic First-Order Methods for Deep Learning
Achraf Bahamou, Donald Goldfarb
We propose a new per-layer adaptive step-size procedure for stochastic first-order optimization methods for minimizing empirical loss functions in deep learning, eliminating the ne…
A Mini-Block Fisher Method for Deep Neural Networks
Achraf Bahamou, Donald Goldfarb, Yi Ren
Deep neural networks (DNNs) are currently predominantly trained using first-order methods. Some of these methods (e.g., Adam, AdaGrad, and RMSprop, and their variants) incorporate…
Tensor Normal Training for Deep Learning Models
Yi Ren, Donald Goldfarb
Despite the predominant use of first-order methods for training deep learning models, second-order methods, and in particular, natural gradient methods, remain of interest because…
Kronecker-factored Quasi-Newton Methods for Deep Learning
Yi Ren, Achraf Bahamou, Donald Goldfarb
Second-order methods have the capability of accelerating optimization by using much richer curvature information than first-order methods. However, most are impractical for deep le…
Practical Quasi-Newton Methods for Training Deep Neural Networks
Donald Goldfarb, Yi Ren, Achraf Bahamou
We consider the development of practical stochastic quasi-Newton, and in particular Kronecker-factored block-diagonal BFGS and L-BFGS methods, for training deep neural networks (DN…
A Dynamic Sampling Adaptive-SGD Method for Machine Learning
Achraf Bahamou, Donald Goldfarb
We propose a stochastic optimization method for minimizing loss functions, expressed as an expected value, that adaptively controls the batch size used in the computation of gradie…