672 citations · 1.6k across the 13 of their papers we have counts for
13 papers
MemWM: Memory-Augmented Text-Based World Model
Yujun Wang, Tao Zhang, Jinhe Bi +9
World models are increasingly used to support planning in agents by predicting how environment states evolve in response to agent actions. Yet fluent next-state predictions can sti…
Policy composition in reinforcement learning via multi-objective policy optimization
Shruti Mishra, Ankit Anand, Jordan Hoffmann +4
We enable reinforcement learning agents to learn successful behavior policies by utilizing relevant pre-existing teacher policies. The teacher policies are introduced as objectives…
Training Compute-Optimal Large Language Models
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch +19
We investigate the optimal model size and number of tokens for training a transformer language model under a given compute budget. We find that current large language models are si…
Unified Scaling Laws for Routed Language Models
Aidan Clark, Diego de las Casas, Aurelia Guy +23
The performance of a language model has been shown to be effectively modeled as a power-law in its parameter count. Here we study the scaling behaviors of Routing Networks: archite…
Scaling Language Models: Methods, Analysis & Insights from Training Gopher
Jack W. Rae, Sebastian Borgeaud, Trevor Cai +77
Language modelling provides a step towards intelligent communication systems by harnessing large repositories of written human knowledge to better predict and understand the world.…
Improving language models by retrieving from trillions of tokens
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann +25
We enhance auto-regressive language models by conditioning on document chunks retrieved from a large corpus, based on local similarity with preceding tokens. With a trillion to…
A Systematic Investigation of Commonsense Knowledge in Large Language Models
Xiang Lorraine Li, Adhiguna Kuncoro, Jordan Hoffmann +3
Language models (LMs) trained on large amounts of data have shown impressive performance on many NLP tasks under the zero-shot and few-shot setup. Here we aim to better understand…
AlgebraNets
Jordan Hoffmann, Simon Schmitt, Simon Osindero +2
Neural networks have historically been built layerwise from the set of functions in , i.e. with activations and weights/parameters represented…
Data-Driven Approach to Encoding and Decoding 3-D Crystal Structures
Jordan Hoffmann, Louis Maestrati, Yoshihide Sawada +3
Generative models have achieved impressive results in many domains including image and text generation. In the natural sciences, generative models have led to rapid progress in aut…
Recurrent Independent Mechanisms
Anirudh Goyal, Alex Lamb, Jordan Hoffmann +4
Learning modular structures which reflect the dynamics of the environment can lead to better generalization and robustness to changes which only affect a few of the underlying caus…
InfoGraph: Unsupervised and Semi-supervised Graph-Level Representation Learning via Mutual Information Maximization
Fan-Yun Sun, Jordan Hoffmann, Vikas Verma +1
This paper studies learning the representations of whole graphs in both unsupervised and semi-supervised scenarios. Graph-level representations are critical in a variety of real-wo…
vGraph: A Generative Model for Joint Community Detection and Node Representation Learning
Fan-Yun Sun, Meng Qu, Jordan Hoffmann +2
This paper focuses on two fundamental tasks of graph analysis: community detection and node representation learning, which capture the global and local structures of graphs, respec…
Machine Learning in a data-limited regime: Augmenting experiments with synthetic data uncovers order in crumpled sheets
Jordan Hoffmann, Yohai Bar-Sinai, Lisa Lee +4
Machine learning has gained widespread attention as a powerful tool to identify structure in complex, high-dimensional data. However, these techniques are ostensibly inapplicable f…