471 citations · 3.4k across the 65 of their papers we have counts for
7 papers · 1 filter
Preventing Gradient Attenuation in Lipschitz Constrained Convolutional Networks
Qiyang Li, Saminul Haque, Cem Anil +3
Lipschitz constraints under L2 norm on deep neural networks are useful for provable adversarial robustness bounds, stable training, and Wasserstein distance estimation. While heuri…
Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapse
James Lucas, George Tucker, Roger Grosse +1
Posterior collapse in Variational Autoencoders (VAEs) arises when the variational posterior distribution closely matches the prior for a subset of latent variables. This paper pres…
Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model
Guodong Zhang, Lala Li, Zachary Nado +5
Increasing the batch size is a popular way to speed up neural network training, but beyond some critical batch size, larger batch sizes yield diminishing returns. In this work, we…
EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasis
Chaoqi Wang, Roger Grosse, Sanja Fidler +1
Reducing the test time resource requirements of a neural network while preserving test accuracy is crucial for running inference on resource-constrained devices. To achieve this go…
Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networks
Guodong Zhang, James Martens, Roger Grosse
Natural gradient descent has proven effective at mitigating the effects of pathological curvature in neural network optimization, but little is known theoretically about its conver…
Functional Variational Bayesian Neural Networks
Shengyang Sun, Guodong Zhang, Jiaxin Shi +1
Variational Bayesian neural networks (BNNs) perform variational inference over weights, but it is difficult to specify meaningful priors and approximate posteriors in a high-dimens…