Publications (29)
Behaviour Suite for Reinforcement Learning
Ian Osband, Yotam Doron, Matteo Hessel +11
This paper introduces the Behaviour Suite for Reinforcement Learning, or bsuite for short. bsuite is a collection of carefully-designed experiments that investigate core capabiliti…
Muesli: Combining Improvements in Policy Optimization
Matteo Hessel, Ivo Danihelka, Fabio Viola +6
We propose a novel policy update that combines regularized policy optimization with model learning as an auxiliary loss. The update (henceforth Muesli) matches MuZero's state-of-th…
Self-Consistent Models and Values
Gregory Farquhar, Kate Baumli, Zita Marinho +4
Learned models of the environment provide reinforcement learning (RL) agents with flexible ways of making predictions about the environment. In particular, models enable planning,…
Multi-task Deep Reinforcement Learning with PopArt
Matteo Hessel, Hubert Soyer, Lasse Espeholt +3
The reinforcement learning community has made great strides in designing algorithms capable of exceeding human performance on specific tasks. These algorithms are mostly trained on…
A Self-Tuning Actor-Critic Algorithm
Tom Zahavy, Zhongwen Xu, Vivek Veeriah +5
Reinforcement learning algorithms are highly sensitive to the choice of hyperparameters, typically requiring significant manual effort to identify hyperparameters that perform well…
When to use parametric models in reinforcement learning?
Hado van Hasselt, Matteo Hessel, John Aslanides
We examine the question of when and how parametric models are most useful in reinforcement learning. In particular, we look at commonalities and differences between parametric mode…
Meta-Gradient Reinforcement Learning with an Objective Discovered Online
Zhongwen Xu, Hado van Hasselt, Matteo Hessel +3
Deep reinforcement learning includes a broad family of algorithms that parameterise an internal representation, such as a value function or policy, by a deep neural network. Each a…
Unicorn: Continual Learning with a Universal, Off-policy Agent
Daniel J. Mankowitz, Augustin ŽÃdek, André Barreto +7
Some real-world domains are best characterized as a single task, but for others this perspective is limiting. Instead, some tasks continually grow in complexity, in tandem with the…
Deep Reinforcement Learning and the Deadly Triad
Hado van Hasselt, Yotam Doron, Florian Strub +3
We know from reinforcement learning theory that temporal difference learning can fail in certain cases. Sutton and Barto (2018) identify a deadly triad of function approximation, b…
Rainbow: Combining Improvements in Deep Reinforcement Learning
Matteo Hessel, Joseph Modayil, Hado van Hasselt +7
The deep reinforcement learning community has made several independent improvements to the DQN algorithm. However, it is unclear which of these extensions are complementary and can…
General non-linear Bellman equations
Hado van Hasselt, John Quan, Matteo Hessel +3
We consider a general class of non-linear Bellman equations. These open up a design space of algorithms that have interesting properties, which has two potential advantages. First,…
DataRater: Meta-Learned Dataset Curation
Dan A. Calian, Gregory Farquhar, Iurii Kemaev +9
The quality of foundation models depends heavily on their training data. Consequently, great efforts have been put into dataset curation. Yet most approaches rely on manual tuning…
Discovering Reinforcement Learning Algorithms
Junhyuk Oh, Matteo Hessel, Wojciech M. Czarnecki +4
Reinforcement learning (RL) algorithms update an agent's parameters according to one of several possible rules, discovered manually through years of research. Automating the discov…
The Predictron: End-To-End Learning and Planning
David Silver, Hado van Hasselt, Matteo Hessel +8
One of the key challenges of artificial intelligence is to learn models that are effective in the context of planning. In this document we introduce the predictron architecture. Th…
Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
Gheorghe Comanici, Eric Bieber, Mike Schaekermann +3431
In this report, we introduce the Gemini 2.X model family: Gemini 2.5 Pro and Gemini 2.5 Flash, as well as our earlier Gemini 2.0 Flash and Flash-Lite models. Gemini 2.5 Pro is our…
Scaling shared model governance via model splitting
Miljan Martic, Jan Leike, Andrew Trask +3
Currently the only techniques for sharing governance of a deep learning model are homomorphic encryption and secure multiparty computation. Unfortunately, neither of these techniqu…
Distributed Prioritized Experience Replay
Dan Horgan, John Quan, David Budden +4
We propose a distributed architecture for deep reinforcement learning at scale, that enables agents to learn effectively from orders of magnitude more data than previously possible…
Podracer architectures for scalable Reinforcement Learning
Matteo Hessel, Manuel Kroiss, Aidan Clark +5
Supporting state-of-the-art AI research requires balancing rapid prototyping, ease of use, and quick iteration, with the ability to deploy experiments at a scale traditionally asso…
Observe and Look Further: Achieving Consistent Performance on Atari
Tobias Pohlen, Bilal Piot, Todd Hester +10
Despite significant advances in the field of deep Reinforcement Learning (RL), today's algorithms still fail to learn human-level policies consistently over a set of diverse tasks…
Emphatic Algorithms for Deep Reinforcement Learning
Ray Jiang, Tom Zahavy, Zhongwen Xu +4
Off-policy learning allows us to learn about possible policies of behavior from experience generated by a different behavior policy. Temporal difference (TD) learning algorithms ca…
Learning values across many orders of magnitude
Hado van Hasselt, Arthur Guez, Matteo Hessel +2
Most learning algorithms are not invariant to the scale of the function that is being approximated. We propose to adaptively normalize the targets used in learning. This is useful…
Discovery of Useful Questions as Auxiliary Tasks
Vivek Veeriah, Matteo Hessel, Zhongwen Xu +6
Arguably, intelligent agents ought to be able to discover their own questions so that in learning answers for them they learn unanticipated useful knowledge and skills; this depart…
Dueling Network Architectures for Deep Reinforcement Learning
Ziyu Wang, Tom Schaul, Matteo Hessel +3
In recent years there have been many successes of using deep representations in reinforcement learning. Still, many of these applications use conventional architectures, such as co…
Expected Eligibility Traces
Hado van Hasselt, Sephora Madjiheurem, Matteo Hessel +3
The question of how to determine which states and actions are responsible for a certain outcome is known as the credit assignment problem and remains a central research question in…
Off-Policy Actor-Critic with Shared Experience Replay
Simon Schmitt, Matteo Hessel, Karen Simonyan
We investigate the combination of actor-critic reinforcement learning algorithms with uniform large-scale experience replay and propose solutions for two challenges: (a) efficient…
On Inductive Biases in Deep Reinforcement Learning
Matteo Hessel, Hado van Hasselt, Joseph Modayil +1
Many deep reinforcement learning algorithms contain inductive biases that sculpt the agent's objective and its interface to the environment. These inductive biases can take many fo…
What Can Learned Intrinsic Rewards Capture?
Zeyu Zheng, Junhyuk Oh, Matteo Hessel +5
The objective of a reinforcement learning agent is to behave so as to maximise the sum of a suitable scalar function of state: the reward. These rewards are typically given and imm…
Transfer in Deep Reinforcement Learning Using Successor Features and Generalised Policy Improvement
André Barreto, Diana Borsa, John Quan +6
The ability to transfer skills across tasks has the potential to scale up reinforcement learning (RL) agents to environments currently out of reach. Recently, a framework based on…
Discovery of Options via Meta-Learned Subgoals
Vivek Veeriah, Tom Zahavy, Matteo Hessel +6
Temporal abstractions in the form of options have been shown to help reinforcement learning (RL) agents learn faster. However, despite prior work on this topic, the problem of disc…