Avoiding Side Effects By Considering Future Tasks
arXiv:2010.07877
Abstract
Designing reward functions is difficult: the designer has to specify what to do (what it means to complete the task) as well as what not to do (side effects that should be avoided while completing the task). To alleviate the burden on the reward designer, we propose an algorithm to automatically generate an auxiliary reward function that penalizes side effects. This auxiliary objective rewards the ability to complete possible future tasks, which decreases if the agent causes side effects during the current task. The future task reward can also give the agent an incentive to interfere with events in the environment that make future tasks less achievable, such as irreversible actions by other agents. To avoid this interference incentive, we introduce a baseline policy that represents a default course of action (such as doing nothing), and use it to filter out future tasks that are not achievable by default. We formally define interference incentives and show that the future task approach with a baseline policy avoids these incentives in the deterministic case. Using gridworld environments that test for side effects and interference, we show that our method avoids interference and is more effective for avoiding side effects than the common approach of penalizing irreversible actions.
arXiv commentPublished in NeurIPS 2020
References in corpus (9)
- Cooperative Inverse Reinforcement Learning
- AI Safety Gridworlds
- Trial without Error: Towards Safe Reinforcement Learning via Human Intervention
- Safe Exploration in Markov Decision Processes
- Combating Reinforcement Learning's Sisyphean Curse with Intrinsic Fear
- Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learning
- Penalizing side effects using stepwise relative reachability
- Low Impact Artificial Intelligences
- Preferences Implicit in the State of the World