526 citations · 3.7k across the 67 of their papers we have counts for
89 papers
Unifying Grokking and Double Descent
Xander Davies, Lauro Langosco, David Krueger
A principled understanding of generalization in deep learning may require unifying disparate observations under a single conceptual framework. Previous work has studied \emph{grokk…
Continuous diffusion for categorical data
Sander Dieleman, Laurent Sartran, Arman Roshannai +11
Diffusion models have quickly become the go-to paradigm for generative modelling of perceptual signals (such as images and sound) through iterative refinement. Their success hinges…
MAD for Robust Reinforcement Learning in Machine Translation
Domenic Donato, Lei Yu, Wang Ling +1
We introduce a new distributed policy gradient algorithm and show that it outperforms existing reward-aware training procedures such as REINFORCE, minimum risk training (MRT) and p…
Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scale
Laurent Sartran, Samuel Barrett, Adhiguna Kuncoro +3
We introduce Transformer Grammars (TGs), a novel class of Transformer language models that combine (i) the expressive power, scalability, and strong performance of Transformers and…
Enabling arbitrary translation objectives with Adaptive Tree Search
Wang Ling, Wojciech Stokowiec, Domenic Donato +4
We introduce an adaptive tree search algorithm, that can find high-scoring outputs under translation models that make no assumptions about the form or structure of the search objec…
Scaling Language Models: Methods, Analysis & Insights from Training Gopher
Jack W. Rae, Sebastian Borgeaud, Trevor Cai +77
Language modelling provides a step towards intelligent communication systems by harnessing large repositories of written human knowledge to better predict and understand the world.…