Penalizing side effects using stepwise relative reachability

Abstract

How can we design safe reinforcement learning agents that avoid unnecessary disruptions to their environment? We show that current approaches to penalizing side effects can introduce bad incentives, e.g. to prevent any irreversible changes in the environment, including the actions of other agents. To isolate the source of such undesirable incentives, we break down side effects penalties into two components: a baseline state and a measure of deviation from this baseline state. We argue that some of these incentives arise from the choice of baseline, and others arise from the choice of deviation measure. We introduce a new variant of the stepwise inaction baseline and a new deviation measure based on relative reachability of states. The combination of these design choices avoids the given undesirable incentives, while simpler baselines and the unreachability measure fail. We demonstrate this empirically by comparing different combinations of baseline and deviation measure choices on a set of gridworld experiments designed to illustrate possible bad incentives.

References in corpus (12)

• Cooperative Inverse Reinforcement Learning
• Deep Q-learning from Demonstrations
• Safe Exploration in Continuous Action Spaces
• Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning
• Variational Intrinsic Control
• AI Safety Gridworlds
• Trial without Error: Towards Safe Reinforcement Learning via Human Intervention
• Safe Exploration in Finite Markov Decision Processes with Gaussian Processes
• Safe Exploration in Markov Decision Processes
• Combating Reinforcement Learning's Sisyphean Curse with Intrinsic Fear
• Low Impact Artificial Intelligences
• Preferences Implicit in the State of the World

Cited by in corpus (10)

• Scalable agent alignment via reward modeling: a research direction
• AI Research Considerations for Human Existential Safety (ARCHES)
• Avoiding Side Effects By Considering Future Tasks
• Reinforcement Learning Under Moral Uncertainty
• SafeLife 1.0: Exploring Side Effects in Complex Environments
• AGI Agent Safety by Iteratively Improving the Utility Function
• Towards Empathic Deep Q-Learning
• Counterfactual Planning in AGI Systems
• Combining Reward Information from Multiple Sources
• Safety Aware Reinforcement Learning (SARL)