REALab: An Embedded Perspective on Tampering
arXiv:2011.08820
Abstract
This paper describes REALab, a platform for embedded agency research in reinforcement learning (RL). REALab is designed to model the structure of tampering problems that may arise in real-world deployments of RL. Standard Markov Decision Process (MDP) formulations of RL and simulated environments mirroring the MDP structure assume secure access to feedback (e.g., rewards). This may be unrealistic in settings where agents are embedded and can corrupt the processes producing feedback (e.g., human supervisors, or an implemented reward function). We describe an alternative Corrupt Feedback MDP formulation and the REALab environment platform, which both avoid the secure feedback assumption. We hope the design of REALab provides a useful perspective on tampering problems, and that the platform may serve as a unit test for the presence of tampering incentives in RL agent designs.
References in corpus (13)
- Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
- Solving Rubik's Cube with a Robot Hand
- RL: Fast Reinforcement Learning via Slow Reinforcement Learning
- Learning to reinforcement learn
- Neural Programmer-Interpreters
- Scalable agent alignment via reward modeling: a research direction
- Effects of Persuasive Dialogues: Testing Bot Identities and Inquiry Strategies
- Neural probabilistic motor primitives for humanoid control
- Risks from Learned Optimization in Advanced Machine Learning Systems
- Good and safe uses of AI Oracles
- AGI Agent Safety by Iteratively Improving the Utility Function
- Corrigibility with Utility Preservation
- Categorizing Wireheading in Partially Embedded Agents