1.4k citations · 8.8k across the 91 of their papers we have counts for
11 papers · 1 filter
Measuring the Effects of Data Parallelism on Neural Network Training
Christopher J. Shallue, Jaehoon Lee, Joseph Antognini +3
Recent hardware developments have dramatically increased the scale of data parallelism available for neural network training. Among the simplest ways to harness next-generation har…
Understanding and correcting pathologies in the training of learned optimizers
Luke Metz, Niru Maheswaranathan, Jeremy Nixon +2
Deep learning has shown that learned functions can dramatically outperform hand-designed functions on perceptual tasks. Analogously, this suggests that learned optimizers may simil…
Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes
Roman Novak, Lechao Xiao, Jaehoon Lee +6
There is a previously identified equivalence between wide fully connected neural networks (FCNs) and Gaussian processes (GPs). This equivalence enables, for instance, test set pred…
Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks
Lechao Xiao, Yasaman Bahri, Jascha Sohl-Dickstein +2
In recent years, state-of-the-art methods in computer vision have utilized increasingly deep convolutional neural network architectures (CNNs), with some of the most successful mod…
PCA of high dimensional random walks with comparison to neural network training
Joseph M. Antognini, Jascha Sohl-Dickstein
One technique to visualize the training of neural networks is to perform PCA on the parameters over the course of training and to project to the subspace spanned by the first few P…
Adversarial Reprogramming of Neural Networks
Gamaleldin F. Elsayed, Ian Goodfellow, Jascha Sohl-Dickstein
Deep neural networks are susceptible to \emph{adversarial} attacks. In computer vision, well-crafted perturbations to images can cause neural networks to make mistakes such as conf…
Guided evolutionary strategies: Augmenting random search with surrogate gradients
Niru Maheswaranathan, Luke Metz, George Tucker +2
Many applications in machine learning require optimizing a function whose true gradient is unknown, but where surrogate gradient information (directions that may be correlated with…
Stochastic natural gradient descent draws posterior samples in function space
Samuel L. Smith, Daniel Duckworth, Semon Rezchikov +2
Recent work has argued that stochastic gradient descent can approximate the Bayesian uncertainty in model parameters near local minima. In this work we develop a similar correspond…
Meta-Learning Update Rules for Unsupervised Representation Learning
Luke Metz, Niru Maheswaranathan, Brian Cheung +1
A major goal of unsupervised learning is to discover data representations that are useful for subsequent tasks, without access to supervised labels during training. Typically, this…
Sensitivity and Generalization in Neural Networks: an Empirical Study
Roman Novak, Yasaman Bahri, Daniel A. Abolafia +2
In practice it is often found that large over-parameterized neural networks generalize better than their smaller counterparts, an observation that appears to conflict with classica…
Adversarial Examples that Fool both Computer Vision and Time-Limited Humans
Gamaleldin F. Elsayed, Shreya Shankar, Brian Cheung +4
Machine learning models are vulnerable to adversarial examples: small changes to images can cause computer vision models to make mistakes such as identifying a school bus as an ost…