1k citations · 2.7k across the 12 of their papers we have counts for
12 papers
Training Compute-Optimal Large Language Models
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch +19
We investigate the optimal model size and number of tokens for training a transformer language model under a given compute budget. We find that current large language models are si…
Unified Scaling Laws for Routed Language Models
Aidan Clark, Diego de las Casas, Aurelia Guy +23
The performance of a language model has been shown to be effectively modeled as a power-law in its parameter count. Here we study the scaling behaviors of Routing Networks: archite…
Scaling Language Models: Methods, Analysis & Insights from Training Gopher
Jack W. Rae, Sebastian Borgeaud, Trevor Cai +77
Language modelling provides a step towards intelligent communication systems by harnessing large repositories of written human knowledge to better predict and understand the world.…
Improving language models by retrieving from trillions of tokens
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann +25
We enhance auto-regressive language models by conditioning on document chunks retrieved from a large corpus, based on local similarity with preceding tokens. With a trillion to…
Podracer architectures for scalable Reinforcement Learning
Matteo Hessel, Manuel Kroiss, Aidan Clark +5
Supporting state-of-the-art AI research requires balancing rapid prototyping, ease of use, and quick iteration, with the ability to deploy experiments at a scale traditionally asso…
Skillful Precipitation Nowcasting using Deep Generative Models of Radar
Suman Ravuri, Karel Lenc, Matthew Willson +17
Precipitation nowcasting, the high-resolution forecasting of precipitation up to two hours ahead, supports the real-world socio-economic needs of many sectors reliant on weather-de…
Transformation-based Adversarial Video Prediction on Large-Scale Data
Pauline Luc, Aidan Clark, Sander Dieleman +4
Recent breakthroughs in adversarial generative modeling have led to models capable of producing video samples of high quality, even on large and complex datasets of real-world vide…
Stabilizing Transformers for Reinforcement Learning
Emilio Parisotto, H. Francis Song, Jack W. Rae +10
Owing to their ability to both effectively integrate information over long time horizons and scale to massive amounts of data, self-attention architectures have recently shown brea…
V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Control
H. Francis Song, Abbas Abdolmaleki, Jost Tobias Springenberg +11
Some of the most successful applications of deep reinforcement learning to challenging domains in discrete and continuous control have used policy gradient methods in the on-policy…
High Fidelity Speech Synthesis with Adversarial Networks
Mikołaj Bińkowski, Jeff Donahue, Sander Dieleman +5
Generative adversarial networks have seen rapid development in recent years and have led to remarkable improvements in generative modelling of images. However, their application in…
Adversarial Video Generation on Complex Datasets
Aidan Clark, Jeff Donahue, Karen Simonyan
Generative models of natural images have progressed towards high fidelity samples by the strong leveraging of scale. We attempt to carry this success to the field of video modeling…
TF-Replicator: Distributed Machine Learning for Researchers
Peter Buchlovsky, David Budden, Dominik Grewe +9
We describe TF-Replicator, a framework for distributed machine learning designed for DeepMind researchers and implemented as an abstraction over TensorFlow. TF-Replicator simplifie…