REALab: An Embedded Perspective on Tampering

Abstract

This paper describes REALab, a platform for embedded agency research in reinforcement learning (RL). REALab is designed to model the structure of tampering problems that may arise in real-world deployments of RL. Standard Markov Decision Process (MDP) formulations of RL and simulated environments mirroring the MDP structure assume secure access to feedback (e.g., rewards). This may be unrealistic in settings where agents are embedded and can corrupt the processes producing feedback (e.g., human supervisors, or an implemented reward function). We describe an alternative Corrupt Feedback MDP formulation and the REALab environment platform, which both avoid the secure feedback assumption. We hope the design of REALab provides a useful perspective on tampering problems, and that the platform may serve as a unit test for the presence of tampering incentives in RL agent designs.

References in corpus (6)

• Solving Rubik's Cube with a Robot Hand
• RL${}^2$: Fast Reinforcement Learning via Slow Reinforcement Learning
• Learning to reinforcement learn
• Effects of Persuasive Dialogues: Testing Bot Identities and Inquiry Strategies
• AGI Agent Safety by Iteratively Improving the Utility Function
• Categorizing Wireheading in Partially Embedded Agents

Cited by in corpus (1)

• Counterfactual Planning in AGI Systems