24 citations · 104 across the 18 of their papers we have counts for
18 papers
Online Conformal Prediction Beyond Feedback
Joar Skalse, Edoardo Pona, Osvaldo Simeone +1
Online conformal prediction (OCP) provides theoretically principled uncertainty quantification for arbitrary black-box classifiers and non-i.i.d. data streams by constructing predi…
Partial Identifiability in Inverse Reinforcement Learning For Agents With Non-Exponential Discounting
Joar Skalse, Alessandro Abate
The aim of inverse reinforcement learning (IRL) is to infer an agent's preferences from observing their behaviour. Usually, preferences are modelled as a reward function, , and…
Partial Identifiability and Misspecification in Inverse Reinforcement Learning
Joar Skalse, Alessandro Abate
The aim of Inverse Reinforcement Learning (IRL) is to infer a reward function from a policy . This problem is difficult, for several reasons. First of all, there are typical…
The Perils of Optimizing Learned Reward Functions: Low Training Error Does Not Guarantee Low Regret
Lukas Fluri, Leon Lang, Alessandro Abate +3
In reinforcement learning, specifying reward functions that capture the intended task can be very challenging. Reward learning aims to address this issue by learning the reward fun…
Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systems
David "davidad" Dalrymple, Joar Skalse, Yoshua Bengio +14
Ensuring that AI systems reliably and robustly avoid harmful or dangerous behaviours is a crucial challenge, especially for AI systems with a high degree of autonomy and general in…
Quantifying the Sensitivity of Inverse Reinforcement Learning to Misspecification
Joar Skalse, Alessandro Abate
Inverse reinforcement learning (IRL) aims to infer an agent's preferences (represented as a reward function ) from their behaviour (represented as a policy ). To do this, we…
On the Limitations of Markovian Rewards to Express Multi-Objective, Risk-Sensitive, and Modal Tasks
Joar Skalse, Alessandro Abate
In this paper, we study the expressivity of scalar, Markovian reward functions in Reinforcement Learning (RL), and identify several limitations to what they can express. Specifical…
Goodhart's Law in Reinforcement Learning
Jacek Karwowski, Oliver Hayman, Xingjian Bai +3
Implementing a reward function that perfectly captures a complex task in the real world is impractical. As a result, it is often appropriate to think of the reward function as a pr…
On The Expressivity of Objective-Specification Formalisms in Reinforcement Learning
Rohan Subramani, Marcus Williams, Max Heitmann +3
Most algorithms in reinforcement learning (RL) require that the objective is formalised with a Markovian reward function. However, it is well-known that certain tasks cannot be exp…
STARC: A General Framework For Quantifying Differences Between Reward Functions
Joar Skalse, Lucy Farnik, Sumeet Ramesh Motwani +3
In order to solve a task using reinforcement learning, it is necessary to first formalise the goal of that task as a reward function. However, for many real-world tasks, it is very…
Lexicographic Multi-Objective Reinforcement Learning
Joar Skalse, Lewis Hammond, Charlie Griffin +1
In this work we introduce reinforcement learning techniques for solving lexicographic multi-objective problems. These are problems that involve multiple reward signals, and where t…
Misspecification in Inverse Reinforcement Learning
Joar Skalse, Alessandro Abate
The aim of Inverse Reinforcement Learning (IRL) is to infer a reward function from a policy . To do this, we need a model of how relates to . In the current literatur…
Defining and Characterizing Reward Hacking
Joar Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov +1
We provide the first formal definition of reward hacking, a phenomenon where optimizing an imperfect proxy reward function leads to poor performance according to the true reward fu…
Invariance in Policy Optimisation and Partial Identifiability in Reward Learning
Joar Skalse, Matthew Farrugia-Roberts, Stuart Russell +2
It is often very challenging to manually design reward functions for complex, real-world tasks. To solve this, one can instead use reward learning to infer a reward function from d…
A General Counterexample to Any Decision Theory and Some Responses
Joar Skalse
In this paper I present an argument and a general schema which can be used to construct a problem case for any decision theory, in a way that could be taken to show that one cannot…
Is SGD a Bayesian sampler? Well, almost
Chris Mingard, Guillermo Valle-Pérez, Joar Skalse +1
Overparameterised deep neural networks (DNNs) are highly expressive and so can, in principle, generate almost any function that fits a training dataset with zero error. The vast ma…
Neural networks are a priori biased towards Boolean functions with low entropy
Chris Mingard, Joar Skalse, Guillermo Valle-Pérez +3
Understanding the inductive bias of neural networks is critical to explaining their ability to generalise. Here, for one of the simplest neural networks -- a single-layer perceptro…
Risks from Learned Optimization in Advanced Machine Learning Systems
Evan Hubinger, Chris van Merwijk, Vladimir Mikulik +2
We analyze the type of learned optimization that occurs when a learned model (such as a neural network) is itself an optimizer - a situation we refer to as mesa-optimization, a neo…