372 citations · 567 across the 16 of their papers we have counts for
13 papers · 1 filter
Replay Can Provably Increase Forgetting
Yasaman Mahdaviyeh, James Lucas, Mengye Ren +3
Continual learning seeks to enable machine learning systems to solve an increasing corpus of tasks sequentially. A critical challenge for continual learning is forgetting, where th…
ATT3D: Amortized Text-to-3D Object Synthesis
Jonathan Lorraine, Kevin Xie, Xiaohui Zeng +7
Text-to-3D modelling has seen exciting progress by combining generative text-to-image models with image-to-3D methods like Neural Radiance Fields. DreamFusion recently achieved hig…
The Calibration Generalization Gap
A. Michael Carrell, Neil Mallinar, James Lucas +1
Calibration is a fundamental property of a good predictive model: it requires that the model predicts correctly in proportion to its confidence. Modern neural networks, however, pr…
Optimizing Data Collection for Machine Learning
Rafid Mahmood, James Lucas, Jose M. Alvarez +2
Modern deep learning systems require huge data sets to achieve impressive performance, but there is little guidance on how much or what kind of data to collect. Over-collecting dat…
Analyzing Monotonic Linear Interpolation in Neural Network Loss Landscapes
James Lucas, Juhan Bae, Michael R. Zhang +3
Linear interpolation between initial neural network parameters and converged parameters after training with stochastic gradient descent (SGD) typically leads to a monotonic decreas…
Probing Few-Shot Generalization with Attributes
Mengye Ren, Eleni Triantafillou, Kuan-Chieh Wang +5
Despite impressive progress in deep learning, generalizing far beyond the training distribution is an important open challenge. In this work, we consider few-shot classification, a…
Regularized linear autoencoders recover the principal components, eventually
Xuchan Bao, James Lucas, Sushant Sachdeva +1
Our understanding of learning input-output relationships with neural nets has improved rapidly in recent years, but little is known about the convergence of the underlying represen…
Preventing Gradient Attenuation in Lipschitz Constrained Convolutional Networks
Qiyang Li, Saminul Haque, Cem Anil +3
Lipschitz constraints under L2 norm on deep neural networks are useful for provable adversarial robustness bounds, stable training, and Wasserstein distance estimation. While heuri…
Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapse
James Lucas, George Tucker, Roger Grosse +1
Posterior collapse in Variational Autoencoders (VAEs) arises when the variational posterior distribution closely matches the prior for a subset of latent variables. This paper pres…
Lookahead Optimizer: k steps forward, 1 step back
Michael R. Zhang, James Lucas, Geoffrey Hinton +1
The vast majority of successful deep neural networks are trained using variants of stochastic gradient descent (SGD) algorithms. Recent attempts to improve SGD can be broadly categ…
Sorting out Lipschitz function approximation
Cem Anil, James Lucas, Roger Grosse
Training neural networks under a strict Lipschitz constraint is useful for provable adversarial robustness, generalization bounds, interpretable gradients, and Wasserstein distance…
Adversarial Distillation of Bayesian Neural Network Posteriors
Kuan-Chieh Wang, Paul Vicol, James Lucas +3
Bayesian neural networks (BNNs) allow us to reason about uncertainty in a principled way. Stochastic Gradient Langevin Dynamics (SGLD) enables efficient BNN learning by drawing sam…
Aggregated Momentum: Stability Through Passive Damping
James Lucas, Shengyang Sun, Richard Zemel +1
Momentum is a simple and widely used trick which allows gradient-based optimizers to pick up speed along low curvature directions. Its performance depends crucially on a damping co…