3.2k citations · 8.8k across the 71 of their papers we have counts for
9 papers · 1 filter
Self-Tuning Stochastic Optimization with Curvature-Aware Gradient Filtering
Ricky T. Q. Chen, Dami Choi, Lukas Balles +2
Standard first-order stochastic optimization algorithms base their updates solely on the average mini-batch gradient, and it has been shown that tracking additional quantities such…
Teaching with Commentaries
Aniruddh Raghu, Maithra Raghu, Simon Kornblith +2
Effective training of deep neural networks can be challenging, and there remain many open questions on how to best learn these models. Recently developed methods to improve neural…
No MCMC for me: Amortized sampling for fast and stable training of energy-based models
Will Grathwohl, Jacob Kelly, Milad Hashemi +3
Energy-Based Models (EBMs) present a flexible and appealing way to represent uncertainty. Despite recent advances, training EBMs on high-dimensional data remains a challenging prob…
A Study of Gradient Variance in Deep Learning
Fartash Faghri, David Duvenaud, David J. Fleet +1
The impact of gradient noise on training deep models is widely acknowledged but not well understood. In this context, we study the distribution of gradients during training. We int…
Learning Differential Equations that are Easy to Solve
Jacob Kelly, Jesse Bettencourt, Matthew James Johnson +1
Differential equations parameterized by neural networks become expensive to solve numerically as training progresses. We propose a remedy that encourages learned dynamics to be eas…
SUMO: Unbiased Estimation of Log Marginal Probability for Latent Variable Models
Yucen Luo, Alex Beatson, Mohammad Norouzi +4
Standard variational lower bounds used to train latent variable models produce biased estimates of most quantities of interest. We introduce an unbiased estimator of the log margin…