Generalization in Text-based Games via Hierarchical Reinforcement Learning

Abstract

Deep reinforcement learning provides a promising approach for text-based games in studying natural language communication between humans and artificial agents. However, the generalization still remains a big challenge as the agents depend critically on the complexity and variety of training tasks. In this paper, we address this problem by introducing a hierarchical framework built upon the knowledge graph-based RL agent. In the high level, a meta-policy is executed to decompose the whole game into a set of subtasks specified by textual goals, and select one of them based on the KG. Then a sub-policy in the low level is executed to conduct goal-conditioned reinforcement learning. We carry out experiments on games with various difficulty levels and show that the proposed method enjoys favorable generalizability.

arXiv comment41 pages, 11 figures, EMNLP2021 Findings

References in corpus (19)

• Prioritized Experience Replay
• Hindsight Experience Replay
• FeUdal Networks for Hierarchical Reinforcement Learning
• Data-Efficient Hierarchical Reinforcement Learning
• Modular Multitask Reinforcement Learning with Policy Sketches
• Why We Need New Evaluation Metrics for NLG
• Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning
• Language as an Abstraction for Hierarchical Deep Reinforcement Learning
• TextWorld: A Learning Environment for Text-based Games
• From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Following
• Learning to Understand Goal Specifications by Modelling Reward
• Learning Dynamic Belief Graphs to Generalize on Text-Based Games
• Graph Constrained Reinforcement Learning for Natural Language Action Spaces
• Hierarchical Reinforcement Learning for Zero-shot Generalization with Subtask Dependencies
• NAIL: A General Interactive Fiction Agent
• BeBold: Exploration Beyond the Boundary of Explored Regions
• Enhancing Text-based Reinforcement Learning Agents with Commonsense Knowledge
• How To Avoid Being Eaten By a Grue: Exploration Strategies for Text-Adventure Agents
• Text-based RL Agents with Commonsense Knowledge: New Challenges, Environments and Baselines