Combating Reinforcement Learning's Sisyphean Curse with Intrinsic Fear
arXiv:1611.01211
Abstract
Many practical environments contain catastrophic states that an optimal agent would visit infrequently or never. Even on toy problems, Deep Reinforcement Learning (DRL) agents tend to periodically revisit these states upon forgetting their existence under a new policy. We introduce intrinsic fear (IF), a learned reward shaping that guards DRL agents against periodic catastrophes. IF agents possess a fear model trained to predict the probability of imminent catastrophe. This score is then used to penalize the Q-learning objective. Our theoretical analysis bounds the reduction in average return due to learning on the perturbed objective. We also prove robustness to classification errors. As a bonus, IF models tend to learn faster, owing to reward shaping. Experiments demonstrate that intrinsic-fear DQNs solve otherwise pathological environments and improve on several Atari games.
References in corpus (2)
Cited by in corpus (24)
- A Survey and Critique of Multiagent Deep Reinforcement Learning
- Exploration in Deep Reinforcement Learning: A Survey
- Go-Explore: a New Approach for Hard-Exploration Problems
- Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain
- Learn What Not to Learn: Action Elimination with Deep Reinforcement Learning
- AI Safety Gridworlds
- Trial without Error: Towards Safe Reinforcement Learning via Human Intervention
- Learning to Walk in the Real World with Minimal Human Effort
- Learning to be Safe: Deep RL with a Safety Critic
- Penalizing side effects using stepwise relative reachability
- Agent-Agnostic Human-in-the-Loop Reinforcement Learning
- Avoiding Side Effects By Considering Future Tasks
- Deep Robust Kalman Filter
- Surprising Negative Results for Generative Adversarial Tree Search
- Safe Deep Reinforcement Learning for Multi-Agent Systems with Continuous Action Spaces
- Generalizing from a few environments in safety-critical reinforcement learning
- Design Space of Behaviour Planning for Autonomous Driving
- Policy Gradient in Partially Observable Environments: Approximation and Convergence
- I'm sorry Dave, I'm afraid I can't do that, Deep Q-learning from forbidden action
- Safer Deep RL with Shallow MCTS: A Case Study in Pommerman
- Evolutionary Computation and AI Safety: Research Problems Impeding Routine and Safe Real-world Application of Evolution
- Safe Reinforcement Learning on Autonomous Vehicles
- Better Safe than Sorry: Evidence Accumulation Allows for Safe Reinforcement Learning
- Evolution of Q Values for Deep Q Learning in Stable Baselines