Learn What Not to Learn: Action Elimination with Deep Reinforcement Learning

Abstract

Learning how to act when there are many available actions in each state is a challenging task for Reinforcement Learning (RL) agents, especially when many of the actions are redundant or irrelevant. In such cases, it is sometimes easier to learn which actions not to take. In this work, we propose the Action-Elimination Deep Q-Network (AE-DQN) architecture that combines a Deep RL algorithm with an Action Elimination Network (AEN) that eliminates sub-optimal actions. The AEN is trained to predict invalid actions, supervised by an external elimination signal provided by the environment. Simulations demonstrate a considerable speedup and added robustness over vanilla DQN in text-based games with over a thousand discrete actions.

References in corpus (17)

• Distributed Representations of Words and Phrases and their Compositionality
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Deep Reinforcement Learning for Dialogue Generation
• A Deep Reinforcement Learning Chatbot
• Towards End-to-End Learning for Dialog State Tracking and Management using Deep Reinforcement Learning
• Continuously Learning Neural Dialogue Management
• Language Understanding for Text-based Games Using Deep Reinforcement Learning
• Graying the black box: Understanding DQNs
• Counting to Explore and Generalize in Text-based Games
• End-to-End Optimization of Task-Oriented Dialogue Model with Deep Reinforcement Learning
• Combating Reinforcement Learning's Sisyphean Curse with Intrinsic Fear
• Hierarchical Decision Making In Electricity Grid Management
• Shallow Updates for Deep Reinforcement Learning
• What can you do with a rock? Affordance extraction via word embeddings
• Visualizing Dynamics: from t-SNE to SEMI-MDPs
• Is a picture worth a thousand words? A Deep Multi-Modal Fusion Architecture for Product Classification in e-commerce
• Using reinforcement learning to learn how to play text-based games

Cited by in corpus (20)

• A Closer Look at Invalid Action Masking in Policy Gradient Algorithms
• Learning Dynamic Belief Graphs to Generalize on Text-Based Games
• An empirical investigation of the challenges of real-world reinforcement learning
• Reinforcement Learning with Combinatorial Actions: An Application to Vehicle Routing
• Deep Reinforcement Learning for Cybersecurity Threat Detection and Protection: A Review
• Generating Adjacency-Constrained Subgoals in Hierarchical Reinforcement Learning
• Interactive Fiction Games: A Colossal Adventure
• Deep Reinforcement Learning with Stacked Hierarchical Attention for Text-based Games
• Strategies for Using Proximal Policy Optimization in Mobile Puzzle Games
• Deep Neural Linear Bandits: Overcoming Catastrophic Forgetting through Likelihood Matching
• Comprehensible Context-driven Text Game Playing
• Integrating Behavior Cloning and Reinforcement Learning for Improved Performance in Dense and Sparse Reward Environments
• I'm sorry Dave, I'm afraid I can't do that, Deep Q-learning from forbidden action
• Constraint-Guided Reinforcement Learning: Augmenting the Agent-Environment-Interaction
• SAT-MARL: Specification Aware Training in Multi-Agent Reinforcement Learning
• Estimating Q(s,s') with Deep Deterministic Dynamics Gradients
• Online Limited Memory Neural-Linear Bandits with Likelihood Matching
• Action Redundancy in Reinforcement Learning
• Playing by the Book: An Interactive Game Approach for Action Graph Extraction from Text
• Learning Natural Language Generation from Scratch