Avoiding Tampering Incentives in Deep RL via Decoupled Approval
arXiv:2011.08827
Abstract
How can we design agents that pursue a given objective when all feedback mechanisms are influenceable by the agent? Standard RL algorithms assume a secure reward function, and can thus perform poorly in settings where agents can tamper with the reward-generating mechanism. We present a principled solution to the problem of learning from influenceable feedback, which combines approval with a decoupled feedback collection procedure. For a natural class of corruption functions, decoupled approval algorithms have aligned incentives both at convergence and for their local updates. Empirically, they also scale to complex 3D environments where tampering is possible.
References in corpus (18)
- Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
- Deep Reinforcement Learning that Matters
- Data-Efficient Hierarchical Reinforcement Learning
- Neural Programmer-Interpreters
- Scalable agent alignment via reward modeling: a research direction
- Trial without Error: Towards Safe Reinforcement Learning via Human Intervention
- Language as an Abstraction for Hierarchical Deep Reinforcement Learning
- Neural probabilistic motor primitives for humanoid control
- Quality of Uncertainty Quantification for Bayesian Neural Network Inference
- From Optimizing Engagement to Measuring Value
- Generalizing Skills with Semi-Supervised Reinforcement Learning
- Deep Reinforcement Learning from Policy-Dependent Human Feedback
- AI safety via debate
- Supervising strong learners by amplifying weak experts
- Risks from Learned Optimization in Advanced Machine Learning Systems
- Incentives for Responsiveness, Instrumental Control and Impact
- Good and safe uses of AI Oracles
- Learning Human Objectives by Evaluating Hypothetical Behavior