7.8k citations · 21.1k across the 144 of their papers we have counts for
14 papers · 1 filter
To Retain or to Adapt? Generalizing Continual Learning
Giulia Lanzillotta, Mandana Samiei, Doina Precup +2
The Continual Learning (CL) literature has long been driven by the goal of mitigating catastrophic forgetting. This objective rests on an often unstated assumption: that a lifelong…
Powerpropagation: A sparsity inducing weight reparameterisation
Jonathan Schwarz, Siddhant M. Jayakumar, Razvan Pascanu +2
The training of sparse neural networks is becoming an increasingly important tool for reducing the computational footprint of models at training and evaluation, as well enabling th…
BYOL works even without batch statistics
Pierre H. Richemond, Jean-Bastien Grill, Florent Altché +8
Bootstrap Your Own Latent (BYOL) is a self-supervised learning approach for image representation. From an augmented view of an image, BYOL trains an online network to predict a tar…
Pointer Graph Networks
Petar Veličković, Lars Buesing, Matthew C. Overlan +3
Graph neural networks (GNNs) are typically applied to static graphs that are assumed to be known upfront. This static input structure is often informed purely by insight of the mac…
Task Agnostic Continual Learning via Meta Learning
Xu He, Jakub Sygnowski, Alexandre Galashov +3
While neural networks are powerful function approximators, they suffer from catastrophic forgetting when the data distribution is not stationary. One particular formalism that stud…
Functional Regularisation for Continual Learning with Gaussian Processes
Michalis K. Titsias, Jonathan Schwarz, Alexander G. de G. Matthews +2
We introduce a framework for Continual Learning (CL) based on Bayesian inference over the function space rather than the parameters of a deep neural network. This method, referred…
Adapting Auxiliary Losses Using Gradient Similarity
Yunshu Du, Wojciech M. Czarnecki, Siddhant M. Jayakumar +3
One approach to deal with the statistical inefficiency of neural networks is to rely on auxiliary losses that help to build useful representations. However, it is not always trivia…
Progress & Compress: A scalable framework for continual learning
Jonathan Schwarz, Jelena Luketina, Wojciech M. Czarnecki +4
We introduce a conceptually simple and scalable framework for continual learning domains where tasks are learned sequentially. Our method is constant in the number of parameters an…
Been There, Done That: Meta-Learning with Episodic Recall
Samuel Ritter, Jane X. Wang, Zeb Kurth-Nelson +4
Meta-learning agents excel at rapidly learning new tasks from open-ended task distributions; yet, they forget what they learn about each task as soon as the next begins. When tasks…
Memory-based Parameter Adaptation
Pablo Sprechmann, Siddhant M. Jayakumar, Jack W. Rae +7
Deep neural networks have excelled on a wide range of problems, from vision to language and game playing. Neural networks very gradually incorporate information into weights as the…
Local minima in training of neural networks
Grzegorz Swirszcz, Wojciech Marian Czarnecki, Razvan Pascanu
There has been a lot of recent interest in trying to characterize the error surface of deep models. This stems from a long standing question. Given that deep networks are highly no…
Natural Neural Networks
Guillaume Desjardins, Karen Simonyan, Razvan Pascanu +1
We introduce Natural Neural Networks, a novel family of algorithms that speed up convergence by adapting their internal representation during training to improve conditioning of th…
On the Number of Linear Regions of Deep Neural Networks
Guido Montúfar, Razvan Pascanu, Kyunghyun Cho +1
We study the complexity of functions computable by deep feedforward neural networks with piecewise linear activations in terms of the symmetries and the number of linear regions th…
Pylearn2: a machine learning research library
Ian J. Goodfellow, David Warde-Farley, Pascal Lamblin +6
Pylearn2 is a machine learning research library. This does not just mean that it is a collection of machine learning algorithms that share a common API; it means that it has been d…