1 citations · 1 across the 1 of their papers we have counts for
4 papers
Self-Tuning Stochastic Optimization with Curvature-Aware Gradient Filtering
Ricky T. Q. Chen, Dami Choi, Lukas Balles +2
Standard first-order stochastic optimization algorithms base their updates solely on the average mini-batch gradient, and it has been shown that tracking additional quantities such…
Gradient Estimation with Stochastic Softmax Tricks
Max B. Paulus, Dami Choi, Daniel Tarlow +2
The Gumbel-Max trick is the basis of many relaxed gradient estimators. These estimators are easy to implement and low variance, but the goal of scaling them comprehensively to larg…
On Empirical Comparisons of Optimizers for Deep Learning
Dami Choi, Christopher J. Shallue, Zachary Nado +3
Selecting an optimizer is a central step in the contemporary deep learning pipeline. In this paper, we demonstrate the sensitivity of optimizer comparisons to the hyperparameter tu…
Faster Neural Network Training with Data Echoing
Dami Choi, Alexandre Passos, Christopher J. Shallue +1
In the twilight of Moore's law, GPUs and other specialized hardware accelerators have dramatically sped up neural network training. However, earlier stages of the training pipeline…