243 citations · 550 across the 9 of their papers we have counts for
8 papers · 1 filter
Top-KAST: Top-K Always Sparse Training
Siddhant M. Jayakumar, Razvan Pascanu, Jack W. Rae +2
Sparse neural networks are becoming increasingly important as the field seeks to improve the performance of existing models by scaling them up, while simultaneously trying to reduc…
Compressive Transformers for Long-Range Sequence Modelling
Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar +1
We present the Compressive Transformer, an attentive sequence model which compresses past memories for long-range sequence learning. We find the Compressive Transformer obtains sta…
Stabilizing Transformers for Reinforcement Learning
Emilio Parisotto, H. Francis Song, Jack W. Rae +10
Owing to their ability to both effectively integrate information over long time horizons and scale to massive amounts of data, self-attention architectures have recently shown brea…
Information asymmetry in KL-regularized RL
Alexandre Galashov, Siddhant M. Jayakumar, Leonard Hasenclever +7
Many real world tasks exhibit rich structure that is repeated across different parts of the state space or in time. In this work we study the possibility of leveraging such repeate…
Meta-learning of Sequential Strategies
Pedro A. Ortega, Jane X. Wang, Mark Rowland +21
In this report we review memory-based meta-learning as a tool for building sample-efficient strategies that learn from past experience to adapt to any task within a target class. O…
Distilling Policy Distillation
Wojciech Marian Czarnecki, Razvan Pascanu, Simon Osindero +3
The transfer of knowledge from one policy to another is an important tool in Deep Reinforcement Learning. This process, referred to as distillation, has been used to great success,…