471 citations · 1.1k across the 28 of their papers we have counts for
11 papers · 1 filter
Eigenvalue Corrected Noisy Natural Gradient
Juhan Bae, Guodong Zhang, Roger Grosse
Variational Bayesian neural networks combine the flexibility of deep learning with Bayesian uncertainty estimation. However, inference procedures for flexible variational posterior…
Sorting out Lipschitz function approximation
Cem Anil, James Lucas, Roger Grosse
Training neural networks under a strict Lipschitz constraint is useful for provable adversarial robustness, generalization bounds, interpretable gradients, and Wasserstein distance…
Three Mechanisms of Weight Decay Regularization
Guodong Zhang, Chaoqi Wang, Bowen Xu +1
Weight decay is one of the standard tricks in the neural network toolbox, but the reasons for its regularization effect are poorly understood, and recent results have cast doubt on…
Reversible Recurrent Neural Networks
Matthew MacKay, Paul Vicol, Jimmy Ba +1
Recurrent neural networks (RNNs) provide state-of-the-art performance in processing sequential data but are memory intensive to train, limiting the flexibility of RNN models which…
A Coordinate-Free Construction of Scalable Natural Gradient
Kevin Luk, Roger Grosse
Most neural networks are trained using first-order optimization methods, which are sensitive to the parameterization of the model. Natural gradient descent is invariant to smooth r…
Adversarial Distillation of Bayesian Neural Network Posteriors
Kuan-Chieh Wang, Paul Vicol, James Lucas +3
Bayesian neural networks (BNNs) allow us to reason about uncertainty in a principled way. Stochastic Gradient Langevin Dynamics (SGLD) enables efficient BNN learning by drawing sam…