Trial without Error: Towards Safe Reinforcement Learning via Human Intervention

Abstract

AI systems are increasingly applied to complex tasks that involve interaction with humans. During training, such systems are potentially dangerous, as they haven't yet learned to avoid actions that could cause serious harm. How can an AI system explore and learn without making a single mistake that harms humans or otherwise causes serious damage? For model-free reinforcement learning, having a human "in the loop" and ready to intervene is currently the only way to prevent all catastrophes. We formalize human intervention for RL and show how to reduce the human labor required by training a supervised learner to imitate the human's intervention decisions. We evaluate this scheme on Atari games, with a Deep RL agent being overseen by a human for four hours. When the class of catastrophes is simple, we are able to prevent all catastrophes without affecting the agent's learning (whereas an RL baseline fails due to catastrophic forgetting). However, this scheme is less successful when catastrophes are more complex: it reduces but does not eliminate catastrophes and the supervised learner fails on adversarial examples found by the agent. Extrapolating to more challenging environments, we show that our implementation would not scale (due to the infeasible amount of human labor required). We outline extensions of the scheme that are necessary if we are to train model-free agents without a single catastrophe.

References in corpus (7)

• Deep reinforcement learning from human preferences
• Generative Adversarial Imitation Learning
• Model-Free Episodic Control
• Transfer from Simulation to Real World through Learning Deep Inverse Dynamics Model
• Learning values across many orders of magnitude
• Combating Reinforcement Learning's Sisyphean Curse with Intrinsic Fear
• Deep Learning of Robotic Tasks without a Simulator using Strong and Weak Human Supervision

Cited by in corpus (25)

• Exploration in Deep Reinforcement Learning: A Survey
• Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain
• Prioritized Experience-based Reinforcement Learning with Human Guidance for Autonomous Driving
• AI Safety Gridworlds
• Software Engineering for AI-Based Systems: A Survey
• OnSlicing: Online End-to-End Network Slicing with Reinforcement Learning
• AI Research Considerations for Human Existential Safety (ARCHES)
• Penalizing side effects using stepwise relative reachability
• Potential Impacts of Smart Homes on Human Behavior: A Reinforcement Learning Approach
• Avoiding Side Effects By Considering Future Tasks
• Automated Lane Change Strategy using Proximal Policy Optimization-based Deep Reinforcement Learning
• Impossibility and Uncertainty Theorems in AI Value Alignment (or why your AGI should not have a utility function)
• Conservative Agency via Attainable Utility Preservation
• Cycle-of-Learning for Autonomous Systems from Human Interaction
• Recent Advances in Leveraging Human Guidance for Sequential Decision-Making Tasks
• Discovering Blind Spots in Reinforcement Learning
• Truthful AI: Developing and governing AI that does not lie
• Improving Safety in Reinforcement Learning Using Model-Based Architectures and Human Intervention
• Active Reinforcement Learning with Monte-Carlo Tree Search
• Formal Language Constraints for Markov Decision Processes
• Play to Grade: Testing Coding Games as Classifying Markov Decision Process
• SAFARI: Safe and Active Robot Imitation Learning with Imagination
• A Mapping of Assurance Techniques for Learning Enabled Autonomous Systems to the Systems Engineering Lifecycle
• Safe Policy Improvement in Constrained Markov Decision Processes
• Avoiding Tampering Incentives in Deep RL via Decoupled Approval