I'm sorry Dave, I'm afraid I can't do that, Deep Q-learning from forbidden action

Abstract

The use of Reinforcement Learning (RL) is still restricted to simulation or to enhance human-operated systems through recommendations. Real-world environments (e.g. industrial robots or power grids) are generally designed with safety constraints in mind implemented in the shape of valid actions masks or contingency controllers. For example, the range of motion and the angles of the motors of a robot can be limited to physical boundaries. Violating constraints thus results in rejected actions or entering in a safe mode driven by an external controller, making RL agents incapable of learning from their mistakes. In this paper, we propose a simple modification of a state-of-the-art deep RL algorithm (DQN), enabling learning from forbidden actions. To do so, the standard Q-learning update is enhanced with an extra safety loss inspired by structured classification. We empirically show that it reduces the number of hit constraints during the learning phase and accelerates convergence to near-optimal policies compared to using standard DQN. Experiments are done on a Visual Grid World Environment and Text-World domain.

Accepted at Internationnal Joint Conference on Neural Networks (IJCNN'2020)

References in corpus (8)

• Deep Q-learning from Demonstrations
• Challenges of Real-World Reinforcement Learning
• Learn What Not to Learn: Action Elimination with Deep Reinforcement Learning
• TextWorld: A Learning Environment for Text-based Games
• Observe and Look Further: Achieving Consistent Performance on Atari
• Social Attention for Autonomous Decision-Making in Dense Traffic
• Combating Reinforcement Learning's Sisyphean Curse with Intrinsic Fear
• Learning Action-Transferable Policy with Action Embedding

Cited by in corpus (4)

• Learning Dynamic Belief Graphs to Generalize on Text-Based Games
• Strategies for Using Proximal Policy Optimization in Mobile Puzzle Games
• Using Offline Data to Speed Up Reinforcement Learning in Procedurally Generated Environments
• SAT-MARL: Specification Aware Training in Multi-Agent Reinforcement Learning