Modular Multitask Reinforcement Learning with Policy Sketches

Abstract

We describe a framework for multitask deep reinforcement learning guided by policy sketches. Sketches annotate tasks with sequences of named subtasks, providing information about high-level structural relationships among tasks but not how to implement them---specifically not providing the detailed guidance used by much previous work on learning policy abstractions for RL (e.g. intermediate rewards, subtask completion signals, or intrinsic motivations). To learn from sketches, we present a model that associates every subtask with a modular subpolicy, and jointly maximizes reward over full task-specific policies by tying parameters across shared subpolicies. Optimization is accomplished via a decoupled actor--critic training objective that facilitates learning common behaviors from multiple dissimilar reward functions. We evaluate the effectiveness of our approach in three environments featuring both discrete and continuous control, and with sparse rewards that can be obtained only after completing a number of high-level subgoals. Experiments show that using our approach to learn policies guided by sketches gives better performance than existing techniques for learning task-specific or shared policies, while naturally inducing a library of interpretable primitive behaviors that can be recombined to rapidly adapt to new tasks.

To appear at ICML 2017

References in corpus (3)

• Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
• Neural Programmer-Interpreters
• Learning to Compose Neural Networks for Question Answering

Cited by in corpus (99)

• A Survey on Multi-Task Learning
• Deep Reinforcement Learning: An Overview
• Multi-Task Learning with Deep Neural Networks: A Survey
• Reward Machines: Exploiting Reward Function Structure in Reinforcement Learning
• Transfer Learning in Deep Reinforcement Learning: A Survey
• Multi-Task Reinforcement Learning with Soft Modularization
• Memorize or generalize? Searching for a compositional RNN in a haystack
• Hierarchical and Interpretable Skill Acquisition in Multi-task Reinforcement Learning
• Environmental drivers of systematicity and generalization in a situated agent
• Composable Planning with Attributes
• TACO: Learning Task Decomposition via Temporal Alignment for Control
• Hierarchical Decision Making by Generating and Following Natural Language Instructions
• Dynamics Learning with Cascaded Variational Inference for Multi-Step Manipulation
• Interactive Grounded Language Acquisition and Generalization in a 2D World
• Diff-DAC: Distributed Actor-Critic for Average Multitask Deep Reinforcement Learning
• Hierarchical Imitation and Reinforcement Learning
• Reinforcement Learning with Competitive Ensembles of Information-Constrained Primitives
• Modular Deep Reinforcement Learning with Temporal Logic Specifications
• Induction and Exploitation of Subgoal Automata for Reinforcement Learning
• A Behavioral Approach to Visual Navigation with Graph Localization Networks
• Multi-Task Reinforcement Learning with Context-based Representations
• Formulation and validation of a car-following model based on deep reinforcement learning
• Learning compositionally through attentive guidance
• LTL2Action: Generalizing LTL Instructions for Multi-Task RL
• Learning Compositional Neural Programs with Recursive Tree Search and Planning
• Neural Task Programming: Learning to Generalize Across Hierarchical Tasks
• Reinforcement Learning with Stochastic Reward Machines
• Neural Task Graphs: Generalizing to Unseen Tasks from a Single Video Demonstration
• Grounding Language to Entities and Dynamics for Generalization in Reinforcement Learning
• Motion Planner Augmented Reinforcement Learning for Robot Manipulation in Obstructed Environments
• Evolving Graphical Planner: Contextual Global Planning for Vision-and-Language Navigation
• Recent Advances in Leveraging Human Guidance for Sequential Decision-Making Tasks
• Sub-policy Adaptation for Hierarchical Reinforcement Learning
• TASKOGRAPHY: Evaluating robot task planning over large 3D scene graphs
• Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation
• M${}^3$RL: Mind-aware Multi-agent Management Reinforcement Learning
• Systematic Generalisation through Task Temporal Logic and Deep Reinforcement Learning
• Extended Markov Games to Learn Multiple Tasks in Multi-Agent Reinforcement Learning
• Enforcing Encoder-Decoder Modularity in Sequence-to-Sequence Models
• Closed Loop Neural-Symbolic Learning via Integrating Neural Perception, Grammar Parsing, and Symbolic Reasoning
• Automata Guided Reinforcement Learning With Demonstrations
• Synthesized Policies for Transfer and Adaptation across Tasks and Environments
• Compositional Transformers for Scene Generation
• CompILE: Compositional Imitation Learning and Execution
• Compositional Reinforcement Learning from Logical Specifications
• Conditional Driving from Natural Language Instructions
• Learning to Synthesize Programs as Interpretable and Generalizable Policies
• Fast Adaptation via Policy-Dynamics Value Functions
• BabyWalk: Going Farther in Vision-and-Language Navigation by Taking Baby Steps
• From Few to More: Large-scale Dynamic Multiagent Curriculum Learning
• Solving Compositional Reinforcement Learning Problems via Task Reduction
• ELLA: Exploration through Learned Language Abstraction
• Active Finite Reward Automaton Inference and Reinforcement Learning Using Queries and Counterexamples
• Multi-Target Embodied Question Answering
• On the Complexity of Exploration in Goal-Driven Navigation
• Ask Your Humans: Using Human Instructions to Improve Generalization in Reinforcement Learning
• Referring Expression Comprehension: A Survey of Methods and Datasets
• Grounding Language for Transfer in Deep Reinforcement Learning
• Encoding formulas as deep networks: Reinforcement learning for zero-shot execution of LTL formulas
• Probing Emergent Semantics in Predictive Agents via Question Answering
• The Logical Options Framework
• Adversarial Skill Chaining for Long-Horizon Robot Manipulation via Terminal State Regularization
• Learning a natural-language to LTL executable semantic parser for grounded robotics
• Skill Transfer in Deep Reinforcement Learning under Morphological Heterogeneity
• Leveraging Human Guidance for Deep Reinforcement Learning Tasks
• Goal Kernel Planning: Linearly-Solvable Non-Markovian Policies for Logical Tasks with Goal-Conditioned Options
• Meta Reinforcement Learning with Autonomous Inference of Subtask Dependencies
• Learning Task Decomposition with Ordered Memory Policy Network
• Hierarchically Decoupled Imitation for Morphological Transfer
• Program Synthesis Guided Reinforcement Learning for Partially Observed Environments
• Block Contextual MDPs for Continual Learning
• Walking with MIND: Mental Imagery eNhanceD Embodied QA
• RLOC: Neurobiologically Inspired Hierarchical Reinforcement Learning Algorithm for Continuous Control of Nonlinear Dynamical Systems
• Fully Distributed Actor-Critic Architecture for Multitask Deep Reinforcement Learning
• Learning compositional programs with arguments and sampling
• Grammar-Based Grounded Lexicon Learning
• SILG: The Multi-environment Symbolic Interactive Language Grounding Benchmark
• Learning Compositional Neural Programs for Continuous Control
• Improving the Universality and Learnability of Neural Programmer-Interpreters with Combinator Abstraction
• Discovering Generalizable Skills via Automated Generation of Diverse Tasks
• Explainable Neural Computation via Stack Neural Module Networks
• Cascade Attribute Network: Decomposing Reinforcement Learning Control Policies using Hierarchical Neural Networks
• Lifelong Reinforcement Learning with Temporal Logic Formulas and Reward Machines
• Interactive Agent Modeling by Learning to Probe
• Fast Task-Adaptation for Tasks Labeled Using Natural Language in Reinforcement Learning
• Efficient Robotic Object Search via HIEM: Hierarchical Policy Learning with Intrinsic-Extrinsic Modeling
• PLOTS: Procedure Learning from Observations using Subtask Structure
• Plan Arithmetic: Compositional Plan Vectors for Multi-Task Control
• Developing cooperative policies for multi-stage tasks
• Biological Blueprints for Next Generation AI Systems
• Integration of Imitation Learning using GAIL and Reinforcement Learning using Task-achievement Rewards via Probabilistic Graphical Model
• Training Transition Policies via Distribution Matching for Complex Tasks
• Reducing the Deployment-Time Inference Control Costs of Deep Reinforcement Learning Agents via an Asymmetric Architecture
• Generalization in Text-based Games via Hierarchical Reinforcement Learning
• Guided Policy Search for Parameterized Skills using Adverbs
• Multitasking Inhibits Semantic Drift
• A Novel Approach to Curiosity and Explainable Reinforcement Learning via Interpretable Sub-Goals
• General AI Challenge - Round One: Gradual Learning
• PICO: Primitive Imitation for COntrol