Publications (43)
Plasticity as the Mirror of Empowerment
David Abel, Michael Bowling, André Barreto +13
Agents are minimally entities that are influenced by their past observations and act to influence future observations. This latter capacity is captured by empowerment, which has se…
A Comparative Analysis of Expected and Distributional Reinforcement Learning
Clare Lyle, Pablo Samuel Castro, Marc G. Bellemare
Since their introduction a year ago, distributional approaches to reinforcement learning (distributional RL) have produced strong results relative to the standard approach which mo…
Understanding Self-Predictive Learning for Reinforcement Learning
Yunhao Tang, Zhaohan Daniel Guo, Pierre Harvey Richemond +13
We study the learning dynamics of self-predictive learning for reinforcement learning, a family of algorithms that learn representations by minimizing the prediction error of their…
Preventing Learning Stagnation in PPO by Scaling to 1 Million Parallel Environments
Michael Beukman, Khimya Khetarpal, Zeyu Zheng +4
An agent's performance stagnating at a suboptimal level is a common problem in deep on-policy RL. Focusing on PPO, we show that plateaus in certain regimes arise not because of kno…
Position: agentic AI orchestration should be Bayes-consistent
Theodore Papamarkou, Pierre Alquier, Matthias Bauer +27
LLMs excel at predictive tasks and complex reasoning tasks, but many high-value deployments rely on decisions under uncertainty, for example, which tool to call, which expert to co…
Mixtures of Experts Unlock Parameter Scaling for Deep RL
Johan Obando-Ceron, Ghada Sokar, Timon Willi +6
The recent rapid progress in (self) supervised learning models is in large part predicted by empirical scaling laws: a model's performance scales proportionally to its size. Analog…
Learning Dynamics and Generalization in Reinforcement Learning
Clare Lyle, Mark Rowland, Will Dabney +2
Solving a reinforcement learning (RL) problem poses two competing challenges: fitting a potentially discontinuous value function, and generalizing well to new observations. In this…
Invariant Causal Prediction for Block MDPs
Amy Zhang, Clare Lyle, Shagun Sodhani +5
Generalization across environments is critical to the successful application of reinforcement learning algorithms to real-world challenges. In this paper, we consider the problem o…
What Can Grokking Teach Us About Learning Under Nonstationarity?
Clare Lyle, Gharda Sokar, Razvan Pascanu +1
In continual learning problems, it is often necessary to overwrite components of a neural network's learned representation in response to changes in the data stream; however, neura…
Normalization and effective learning rates in reinforcement learning
Clare Lyle, Zeyu Zheng, Khimya Khetarpal +4
Normalization layers have recently experienced a renaissance in the deep reinforcement learning and continual learning literature, with several works highlighting diverse benefits…
Understanding and Preventing Capacity Loss in Reinforcement Learning
Clare Lyle, Mark Rowland, Will Dabney
The reinforcement learning (RL) problem is rife with sources of non-stationarity, making it a notoriously difficult problem domain for the application of neural networks. We identi…
PsiPhi-Learning: Reinforcement Learning with Demonstrations using Successor Features and Inverse Temporal Difference Learning
Angelos Filos, Clare Lyle, Yarin Gal +3
We study reinforcement learning (RL) with no-reward demonstrations, a setting in which an RL agent has access to additional data from the interaction of other agents with the same…
DARTS without a Validation Set: Optimizing the Marginal Likelihood
Miroslav Fil, Binxin Ru, Clare Lyle +1
The success of neural architecture search (NAS) has historically been limited by excessive compute requirements. While modern weight-sharing NAS methods such as DARTS are able to f…
DiscoBAX: Discovery of Optimal Intervention Sets in Genomic Experiment Design
Clare Lyle, Arash Mehrjou, Pascal Notin +4
The discovery of therapeutics to treat genetically-driven pathologies relies on identifying genes involved in the underlying disease mechanisms. Existing approaches search over the…
Balancing Expressivity and Robustness: Constrained Rational Activations for Reinforcement Learning
RafaÅ Surdej, MichaÅ Bortkiewicz, Alex Lewandowski +2
Trainable activation functions, whose parameters are optimized alongside network weights, offer increased expressivity compared to fixed activation functions. Specifically, trainab…
Robustness to Pruning Predicts Generalization in Deep Neural Networks
Lorenz Kuhn, Clare Lyle, Aidan N. Gomez +2
Existing generalization measures that aim to capture a model's simplicity based on parameter counts or norms fail to explain generalization in overparameterized deep neural network…
Generalization Through the Lens of Learning Dynamics
Clare Lyle
A machine learning (ML) system must learn not only to match the output of a target function on a training set, but also to generalize to novel situations in order to yield accurate…
Deep Reinforcement Learning with Plasticity Injection
Evgenii Nikishin, Junhyuk Oh, Georg Ostrovski +4
A growing body of evidence suggests that neural networks employed in deep reinforcement learning (RL) gradually lose their plasticity, the ability to learn from new data; however,…
On the Benefits of Invariance in Neural Networks
Clare Lyle, Mark van der Wilk, Marta Kwiatkowska +2
Many real world data analysis problems exhibit invariant structure, and models that take advantage of this structure have shown impressive empirical performance, particularly in de…
Weight Clipping for Deep Continual and Reinforcement Learning
Mohamed Elsayed, Qingfeng Lan, Clare Lyle +1
Many failures in deep continual and reinforcement learning are associated with increasing magnitudes of the weights, making them hard to change and potentially causing overfitting.…
Fine-Tuned In-Context Learners for Efficient Adaptation
Jorg Bornschein, Clare Lyle, Yazhe Li +3
When adapting large language models (LLMs) to a specific downstream task, two primary approaches are commonly employed: (1) prompt engineering, often with in-context few-shot learn…
Unpacking Information Bottlenecks: Unifying Information-Theoretic Objectives in Deep Learning
Andreas Kirsch, Clare Lyle, Yarin Gal
The Information Bottleneck principle offers both a mechanism to explain how deep neural networks train and generalize, as well as a regularized objective with which to train models…
GAN Q-learning
Thang Doan, Bogdan Mazoure, Clare Lyle
Distributional reinforcement learning (distributional RL) has seen empirical success in complex Markov Decision Processes (MDPs) in the setting of nonlinear function approximation.…
Some Counterexamples for Compatible Triangulations
Cody Barnson, Dawn Chandler, Qiao Chen +13
We consider the conjecture by Aichholzer, Aurenhammer, Hurtado, and Krasser that any two points sets with the same cardinality and the same size convex hull can be triangulated in…
On The Effect of Auxiliary Tasks on Representation Dynamics
Clare Lyle, Mark Rowland, Georg Ostrovski +1
While auxiliary tasks play a key role in shaping the representations learnt by reinforcement learning agents, much is still unknown about the mechanisms through which this is achie…
Optimizers Qualitatively Alter Solutions And We Should Leverage This
Razvan Pascanu, Clare Lyle, Ionut-Vlad Modoranu +6
Due to the nonlinear nature of Deep Neural Networks (DNNs), one can not guarantee convergence to a unique global minimum of the loss when using optimizers relying only on local inf…
The Statistical Benefits of Quantile Temporal-Difference Learning for Value Estimation
Mark Rowland, Yunhao Tang, Clare Lyle +3
We study the problem of temporal-difference-based policy evaluation in reinforcement learning. In particular, we analyse the use of a distributional reinforcement learning algorith…
The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigation
Miles Brundage, Shahar Avin, Jack Clark +23
This report surveys the landscape of potential security threats from malicious uses of AI, and proposes ways to better forecast, prevent, and mitigate these threats. After analyzin…
Speedy Performance Estimation for Neural Architecture Search
Binxin Ru, Clare Lyle, Lisa Schut +3
Reliable yet efficient evaluation of generalisation performance of a proposed architecture is crucial to the success of neural architecture search (NAS). Traditional approaches fac…
Near-Minimax-Optimal Distributional Reinforcement Learning with a Generative Model
Mark Rowland, Li Kevin Wenliang, Rémi Munos +3
We propose a new algorithm for model-based distributional reinforcement learning (RL), and prove that it is minimax-optimal for approximating return distributions with a generative…
Frequency and Generalisation of Periodic Activation Functions in Reinforcement Learning
Augustine N. Mavor-Parker, Matthew J. Sargent, Caswell Barry +2
Periodic activation functions, often referred to as learned Fourier features have been widely demonstrated to improve sample efficiency and stability in a variety of deep RL algori…
Slow and Steady Wins the Race: Maintaining Plasticity with Hare and Tortoise Networks
Hojoon Lee, Hyeonseo Cho, Hyunseung Kim +4
This study investigates the loss of generalization ability in neural networks, revisiting warm-starting experiments from Ash & Adams. Our empirical analysis reveals that common met…
A Bayesian Perspective on Training Speed and Model Selection
Clare Lyle, Lisa Schut, Binxin Ru +2
We take a Bayesian perspective to illustrate a connection between training speed and the marginal likelihood in linear models. This provides two major insights: first, that a measu…
Understanding plasticity in neural networks
Clare Lyle, Zeyu Zheng, Evgenii Nikishin +3
Plasticity, the ability of a neural network to quickly change its predictions in response to new information, is essential for the adaptability and robustness of deep reinforcement…
Agency Is Frame-Dependent
David Abel, André Barreto, Michael Bowling +13
Agency is a system's capacity to steer outcomes toward a goal, and is a central topic of study across biology, philosophy, cognitive science, and artificial intelligence. Determini…
Provable Guarantees on the Robustness of Decision Rules to Causal Interventions
Benjie Wang, Clare Lyle, Marta Kwiatkowska
Robustness of decision rules to shifts in the data-generating process is crucial to the successful deployment of decision-making systems. Such shifts can be viewed as interventions…
Non-Stationary Learning of Neural Networks with Automatic Soft Parameter Reset
Alexandre Galashov, Michalis K. Titsias, András György +4
Neural networks are traditionally trained under the assumption that data come from a stationary distribution. However, settings which violate this assumption are becoming more popu…
A Geometric Perspective on Optimal Representations for Reinforcement Learning
Marc G. Bellemare, Will Dabney, Robert Dadashi +6
We propose a new perspective on representation learning in reinforcement learning based on geometric properties of the space of value functions. We leverage this perspective to pro…
A Unifying Framework for Action-Conditional Self-Predictive Reinforcement Learning
Khimya Khetarpal, Zhaohan Daniel Guo, Bernardo Avila Pires +7
Learning a good representation is a crucial challenge for Reinforcement Learning (RL) agents. Self-predictive learning provides means to jointly learn a latent representation and d…
Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learning
Jannik Kossen, Neil Band, Clare Lyle +3
We challenge a common assumption underlying most supervised deep learning: that a model makes a prediction depending only on its parameters and the features of a single input. To t…
Disentangling the Causes of Plasticity Loss in Neural Networks
Clare Lyle, Zeyu Zheng, Khimya Khetarpal +4
Underpinning the past decades of work on the design, initialization, and optimization of neural networks is a seemingly innocuous assumption: that the network is trained on a \text…
Vision-Language Models as a Source of Rewards
Kate Baumli, Satinder Baveja, Feryal Behbahani +24
Building generalist agents that can accomplish many goals in rich open-ended environments is one of the research frontiers for reinforcement learning. A key limiting factor for bui…
V-Simba: Unleashing the Architectural Potential of RL in Visual Continuous Control
Donghu Kim, Youngdo Lee, Hojoon Lee +6
Improving sample efficiency remains a core challenge in reinforcement learning (RL), especially in real-world settings like robotics, where data collection is costly. This challeng…