7.8k citations · 21.1k across the 144 of their papers we have counts for
12 papers · 1 filter
Powerpropagation: A sparsity inducing weight reparameterisation
Jonathan Schwarz, Siddhant M. Jayakumar, Razvan Pascanu +2
The training of sparse neural networks is becoming an increasingly important tool for reducing the computational footprint of models at training and evaluation, as well enabling th…
Wide Neural Networks Forget Less Catastrophically
Seyed Iman Mirzadeh, Arslan Chaudhry, Dong Yin +4
A primary focus area in continual learning research is alleviating the "catastrophic forgetting" problem in neural networks by designing new algorithms that are more robust to the…
On the Role of Optimization in Double Descent: A Least Squares Study
Ilja Kuzborskij, Csaba Szepesvári, Omar Rivasplata +2
Empirically it has been observed that the performance of deep neural networks steadily improves as we increase model size, contradicting the classical view on overfitting and gener…
Reasoning-Modulated Representations
Petar Veličković, Matko Bošnjak, Thomas Kipf +4
Neural networks leverage robust internal representations in order to generalise. Learning them is difficult, and often requires a large training set that covers the data distributi…
Task-agnostic Continual Learning with Hybrid Probabilistic Models
Polina Kirichenko, Mehrdad Farajtabar, Dushyant Rao +6
Learning new tasks continuously without forgetting on a constantly changing data distribution is essential for real-world problems but extremely challenging for modern deep learnin…
Top-KAST: Top-K Always Sparse Training
Siddhant M. Jayakumar, Razvan Pascanu, Jack W. Rae +2
Sparse neural networks are becoming increasingly important as the field seeks to improve the performance of existing models by scaling them up, while simultaneously trying to reduc…
Test Sample Accuracy Scales with Training Sample Density in Neural Networks
Xu Ji, Razvan Pascanu, Devon Hjelm +2
Intuitively, one would expect accuracy of a trained neural network's prediction on test samples to correlate with how densely the samples are surrounded by seen training samples in…
A study on the plasticity of neural networks
Tudor Berariu, Wojciech Czarnecki, Soham De +4
One aim shared by multiple settings, such as continual learning or transfer learning, is to leverage previously acquired knowledge to converge faster on the current task. Usually t…
Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspective
Florin Gogianu, Tudor Berariu, Mihaela Rosca +3
Most of the recent deep reinforcement learning advances take an RL-centric perspective and focus on refinements of the training objective. We diverge from this view and show we can…
Drawing Multiple Augmentation Samples Per Image During Training Efficiently Decreases Test Error
Stanislav Fort, Andrew Brock, Razvan Pascanu +2
In computer vision, it is standard practice to draw a single sample from the data augmentation procedure for each unique image in the mini-batch. However recent work has suggested…
Continual World: A Robotic Benchmark For Continual Reinforcement Learning
Maciej Wołczyk, Michał Zając, Razvan Pascanu +2
Continual learning (CL) -- the ability to continuously learn, building on previously acquired knowledge -- is a natural requirement for long-lived autonomous reinforcement learning…
Regularized Behavior Value Estimation
Caglar Gulcehre, Sergio Gómez Colmenarejo, Ziyu Wang +7
Offline reinforcement learning restricts the learning process to rely only on logged-data without access to an environment. While this enables real-world applications, it also pose…