Learning Dynamic Belief Graphs to Generalize on Text-Based Games

Abstract

Playing text-based games requires skills in processing natural language and sequential decision making. Achieving human-level performance on text-based games remains an open challenge, and prior research has largely relied on hand-crafted structured representations and heuristics. In this work, we investigate how an agent can plan and generalize in text-based games using graph-structured representations learned end-to-end from raw text. We propose a novel graph-aided transformer agent (GATA) that infers and updates latent belief graphs during planning to enable effective action selection by capturing the underlying game dynamics. GATA is trained using a combination of reinforcement and self-supervised learning. Our work demonstrates that the learned graph-based representations help agents converge to better policies than their text-only counterparts and facilitate effective generalization across game configurations. Experiments on 500+ unique games from the TextWorld suite show that our best agent outperforms text-based baselines by an average of 24.2%.

Bug fixed in Table 1

References in corpus (5)

• Layer Normalization
• Rainbow: Combining Improvements in Deep Reinforcement Learning
• Graph Constrained Reinforcement Learning for Natural Language Action Spaces
• NAIL: A General Interactive Fiction Agent
• Building Dynamic Knowledge Graphs from Text-based Games

Cited by in corpus (16)

• Enhancing Text-based Reinforcement Learning Agents with Commonsense Knowledge
• Building Human-like Communicative Intelligence: A Grounded Perspective
• Zero-Shot Learning with Common Sense Knowledge Graphs
• How to Query Language Models?
• WordCraft: An Environment for Benchmarking Commonsense Agents
• XLVIN: eXecuted Latent Value Iteration Nets
• What Would Jiminy Cricket Do? Towards Agents That Behave Morally
• Towards Socially Intelligent Agents with Mental State Transition and Human Utility
• Process-Level Representation of Scientific Protocols with Interactive Annotation
• Interactive Fiction Game Playing as Multi-Paragraph Reading Comprehension with Reinforcement Learning
• Bootstrapped Q-learning with Context Relevant Observation Pruning to Generalize in Text-based Games
• A survey of benchmarking frameworks for reinforcement learning
• Neuro-Symbolic Reinforcement Learning with First-Order Logic
• Interactive Machine Comprehension with Dynamic Knowledge Graphs
• Generalization in Text-based Games via Hierarchical Reinforcement Learning
• Neural Algorithmic Reasoners are Implicit Planners