Graph Constrained Reinforcement Learning for Natural Language Action Spaces

Abstract

Interactive Fiction games are text-based simulations in which an agent interacts with the world purely through natural language. They are ideal environments for studying how to extend reinforcement learning agents to meet the challenges of natural language understanding, partial observability, and action generation in combinatorially-large text-based action spaces. We present KG-A2C, an agent that builds a dynamic knowledge graph while exploring and generates actions using a template-based action space. We contend that the dual uses of the knowledge graph to reason about game state and to constrain natural language generation are the keys to scalable exploration of combinatorially large natural language actions. Results across a wide variety of IF games show that KG-A2C outperforms current IF agents despite the exponential increase in action space size.

Accepted to ICLR 2020

References in corpus (6)

• Efficient Estimation of Word Representations in Vector Space
• Neural Machine Translation by Jointly Learning to Align and Translate
• TextWorld: A Learning Environment for Text-based Games
• NAIL: A General Interactive Fiction Agent
• Interactive Language Learning by Question Answering
• Story Ending Generation with Incremental Encoding and Commonsense Knowledge

Cited by in corpus (16)

• Learning Dynamic Belief Graphs to Generalize on Text-Based Games
• Reinforcement Learning for Generative AI: State of the Art, Opportunities and Open Research Challenges
• ALFWorld: Aligning Text and Embodied Environments for Interactive Learning
• How to Avoid Being Eaten by a Grue: Structured Exploration Strategies for Textual Worlds
• Deep Reinforcement Learning with Stacked Hierarchical Attention for Text-based Games
• Playing Text-Based Games with Common Sense
• How To Avoid Being Eaten By a Grue: Exploration Strategies for Text-Adventure Agents
• Toward Automated Quest Generation in Text-Adventure Games
• What Would Jiminy Cricket Do? Towards Agents That Behave Morally
• Reading and Acting while Blindfolded: The Need for Semantics in Text Game Agents
• Deriving Commonsense Inference Tasks from Interactive Fictions
• Detecting and Adapting to Novelty in Games
• Learning Knowledge Graph-based World Models of Textual Environments
• Interactive Fiction Game Playing as Multi-Paragraph Reading Comprehension with Reinforcement Learning
• Modeling Worlds in Text
• Generalization in Text-based Games via Hierarchical Reinforcement Learning