243 citations · 731 across the 15 of their papers we have counts for
15 papers
One-Step Diffusion Distillation via Deep Equilibrium Models
Zhengyang Geng, Ashwini Pokle, J. Zico Kolter
Diffusion models excel at producing high-quality samples but naively require hundreds of iterations, prompting multiple attempts to distill the generation process into a faster net…
Scaling Language Models: Methods, Analysis & Insights from Training Gopher
Jack W. Rae, Sebastian Borgeaud, Trevor Cai +77
Language modelling provides a step towards intelligent communication systems by harnessing large repositories of written human knowledge to better predict and understand the world.…
Powerpropagation: A sparsity inducing weight reparameterisation
Jonathan Schwarz, Siddhant M. Jayakumar, Razvan Pascanu +2
The training of sparse neural networks is becoming an increasingly important tool for reducing the computational footprint of models at training and evaluation, as well enabling th…
Top-KAST: Top-K Always Sparse Training
Siddhant M. Jayakumar, Razvan Pascanu, Jack W. Rae +2
Sparse neural networks are becoming increasingly important as the field seeks to improve the performance of existing models by scaling them up, while simultaneously trying to reduc…
Perception-Prediction-Reaction Agents for Deep Reinforcement Learning
Adam Stooke, Valentin Dalibard, Siddhant M. Jayakumar +2
We introduce a new recurrent agent architecture and associated auxiliary losses which improve reinforcement learning in partially observable tasks requiring long-term memory. We em…
Compressive Transformers for Long-Range Sequence Modelling
Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar +1
We present the Compressive Transformer, an attentive sequence model which compresses past memories for long-range sequence learning. We find the Compressive Transformer obtains sta…