3.2k citations · 8.8k across the 71 of their papers we have counts for
6 papers · 1 filter
Meta-Learning to Improve Pre-Training
Aniruddh Raghu, Jonathan Lorraine, Simon Kornblith +2
Pre-training (PT) followed by fine-tuning (FT) is an effective method for training neural networks, and has led to significant performance improvements in many domains. PT can inco…
Getting to the Point. Index Sets and Parallelism-Preserving Autodiff for Pointful Array Programming
Adam Paszke, Daniel Johnson, David Duvenaud +5
We present a novel programming language design that attempts to combine the clarity and safety of high-level functional languages with the efficiency and parallelism of low-level n…
Complex Momentum for Optimization in Games
Jonathan Lorraine, David Acuna, Paul Vicol +1
We generalize gradient descent with momentum for optimization in differentiable games to have complex-valued momentum. We give theoretical motivation for our method by proving conv…
Infinitely Deep Bayesian Neural Networks with Stochastic Differential Equations
Winnie Xu, Ricky T. Q. Chen, Xuechen Li +1
We perform scalable approximate inference in continuous-depth Bayesian neural networks. In this model class, uncertainty about separate weights in each layer gives hidden units tha…
Oops I Took A Gradient: Scalable Sampling for Discrete Distributions
Will Grathwohl, Kevin Swersky, Milad Hashemi +2
We propose a general and scalable approximate sampling strategy for probabilistic models with discrete variables. Our approach uses gradients of the likelihood function with respec…
Cost-Efficient Online Hyperparameter Optimization
Jingkang Wang, Mengye Ren, Ilija Bogunovic +2
Recent work on hyperparameters optimization (HPO) has shown the possibility of training certain hyperparameters together with regular parameters. However, these online HPO algorith…