Learning Dynamic Belief Graphs to Generalize on Text-Based Games

Abstract

Playing text-based games requires skills in processing natural language and sequential decision making. Achieving human-level performance on text-based games remains an open challenge, and prior research has largely relied on hand-crafted structured representations and heuristics. In this work, we investigate how an agent can plan and generalize in text-based games using graph-structured representations learned end-to-end from raw text. We propose a novel graph-aided transformer agent (GATA) that infers and updates latent belief graphs during planning to enable effective action selection by capturing the underlying game dynamics. GATA is trained using a combination of reinforcement and self-supervised learning. Our work demonstrates that the learned graph-based representations help agents converge to better policies than their text-only counterparts and facilitate effective generalization across game configurations. Experiments on 500+ unique games from the TextWorld suite show that our best agent outperforms text-based baselines by an average of 24.2%.

arXiv commentBug fixed in Table 1

References in corpus (20)

• Representation Learning with Contrastive Predictive Coding
• Learning deep representations by mutual information estimation and maximization
• Bidirectional Attention Flow for Machine Comprehension
• Highway Networks
• Layer Normalization
• Rainbow: Combining Improvements in Deep Reinforcement Learning
• Neural Relational Inference for Interacting Systems
• Go-Explore: a New Approach for Hard-Exploration Problems
• Learning to Poke by Poking: Experiential Learning of Intuitive Physics
• Learn What Not to Learn: Action Elimination with Deep Reinforcement Learning
• Language Understanding for Text-based Games Using Deep Reinforcement Learning
• TextWorld: A Learning Environment for Text-based Games
• Counting to Explore and Generalize in Text-based Games
• Building Dynamic Knowledge Graphs from Text using Machine Reading Comprehension
• Graph Constrained Reinforcement Learning for Natural Language Action Spaces
• NAIL: A General Interactive Fiction Agent
• GLoMo: Unsupervisedly Learned Relational Graphs as Transferable Representations
• Does Order Matter? An Empirical Study on Generating Multiple Keyphrases as a Sequence
• Building Dynamic Knowledge Graphs from Text-based Games
• I'm sorry Dave, I'm afraid I can't do that, Deep Q-learning from forbidden action

Cited by in corpus (16)

• Enhancing Text-based Reinforcement Learning Agents with Commonsense Knowledge
• Building Human-like Communicative Intelligence: A Grounded Perspective
• Zero-Shot Learning with Common Sense Knowledge Graphs
• How to Query Language Models?
• WordCraft: An Environment for Benchmarking Commonsense Agents
• XLVIN: eXecuted Latent Value Iteration Nets
• What Would Jiminy Cricket Do? Towards Agents That Behave Morally
• Towards Socially Intelligent Agents with Mental State Transition and Human Utility
• Process-Level Representation of Scientific Protocols with Interactive Annotation
• Interactive Fiction Game Playing as Multi-Paragraph Reading Comprehension with Reinforcement Learning
• Bootstrapped Q-learning with Context Relevant Observation Pruning to Generalize in Text-based Games
• A survey of benchmarking frameworks for reinforcement learning
• Neuro-Symbolic Reinforcement Learning with First-Order Logic
• Interactive Machine Comprehension with Dynamic Knowledge Graphs
• Generalization in Text-based Games via Hierarchical Reinforcement Learning
• Neural Algorithmic Reasoners are Implicit Planners