Language as an Abstraction for Hierarchical Deep Reinforcement Learning

Abstract

Solving complex, temporally-extended tasks is a long-standing problem in reinforcement learning (RL). We hypothesize that one critical element of solving such problems is the notion of compositionality. With the ability to learn concepts and sub-skills that can be composed to solve longer tasks, i.e. hierarchical RL, we can acquire temporally-extended behaviors. However, acquiring effective yet general abstractions for hierarchical RL is remarkably challenging. In this paper, we propose to use language as the abstraction, as it provides unique compositional structure, enabling fast learning and combinatorial generalization, while retaining tremendous flexibility, making it suitable for a variety of problems. Our approach learns an instruction-following low-level policy and a high-level policy that can reuse abstractions across tasks, in essence, permitting agents to reason using structured language. To study compositional task learning, we introduce an open-source object interaction environment built using the MuJoCo physics engine and the CLEVR engine. We find that, using our approach, agents can learn to solve to diverse, temporally-extended tasks such as object sorting and multi-object rearrangement, including from raw pixel observations. Our analysis reveals that the compositional nature of language is critical for learning diverse sub-skills and systematically generalizing to new sub-skills in comparison to non-compositional abstractions that use the same supervision.

arXiv commentPublished in Neural Information Processing Systems (NeurIPS) 2019; Supplementary materials: https://sites.google.com/view/hal-demo

Cited by in corpus (26)

• The NetHack Learning Environment
• ALFWorld: Aligning Text and Embodied Environments for Interactive Learning
• Compositional Generalization by Learning Analytical Expressions
• Learning Vision-Guided Quadrupedal Locomotion End-to-End with Cross-Modal Transformers
• Multi-Task Reinforcement Learning with Context-based Representations
• LTL2Action: Generalizing LTL Instructions for Multi-Task RL
• Grounding Language to Autonomously-Acquired Skills via Goal Generation
• Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation
• Solving Compositional Reinforcement Learning Problems via Task Reduction
• Example-Driven Model-Based Reinforcement Learning for Solving Long-Horizon Visuomotor Tasks
• ELLA: Exploration through Learned Language Abstraction
• Language Grounding through Social Interactions and Curiosity-Driven Multi-Goal Learning
• Ask Your Humans: Using Human Instructions to Improve Generalization in Reinforcement Learning
• Following Instructions by Imagining and Reaching Visual Goals
• Hierarchical principles of embodied reinforcement learning: A review
• Catalytic Role Of Noise And Necessity Of Inductive Biases In The Emergence Of Compositional Communication
• Language Conditioned Imitation Learning over Unstructured Data
• Interactive Learning from Activity Description
• Avoiding Tampering Incentives in Deep RL via Decoupled Approval
• On (Emergent) Systematic Generalisation and Compositionality in Visual Referential Games with Straight-Through Gumbel-Softmax Estimator
• DAIR: Disentangled Attention Intrinsic Regularization for Safe and Efficient Bimanual Manipulation
• Generalization in Text-based Games via Hierarchical Reinforcement Learning
• Estimating Disentangled Belief about Hidden State and Hidden Task for Meta-RL
• Multitasking Inhibits Semantic Drift
• Teachable Reinforcement Learning via Advice Distillation
• Generalizable and Explainable Dialogue Generation via Explicit Action Learning