Solving Compositional Reinforcement Learning Problems via Task Reduction

Abstract

We propose a novel learning paradigm, Self-Imitation via Reduction (SIR), for solving compositional reinforcement learning problems. SIR is based on two core ideas: task reduction and self-imitation. Task reduction tackles a hard-to-solve task by actively reducing it to an easier task whose solution is known by the RL agent. Once the original hard task is successfully solved by task reduction, the agent naturally obtains a self-generated solution trajectory to imitate. By continuously collecting and imitating such demonstrations, the agent is able to progressively expand the solved subspace in the entire task space. Experiment results show that SIR can significantly accelerate and improve learning on a variety of challenging sparse-reward continuous-control problems with compositional structures. Code and videos are available at https://sites.google.com/view/sir-compositional.

Project website link updated

References in corpus (18)

• Practical Bayesian Optimization of Machine Learning Algorithms
• Language Models are Few-Shot Learners
• Dota 2 with Large Scale Deep Reinforcement Learning
• FeUdal Networks for Hierarchical Reinforcement Learning
• Data-Efficient Hierarchical Reinforcement Learning
• Modular Multitask Reinforcement Learning with Policy Sketches
• Freeze-Thaw Bayesian Optimization
• Language as an Abstraction for Hierarchical Deep Reinforcement Learning
• Skew-Fit: State-Covering Self-Supervised Reinforcement Learning
• Planning with Goal-Conditioned Policies
• Automatically Composing Representation Transformations as a Means for Generalization
• Unsupervised Control Through Non-Parametric Discriminative Rewards
• Compositional Planning Using Optimal Option Models
• Meta-Learning MCMC Proposals
• Teacher algorithms for curriculum learning of Deep RL in continuously parameterized environments
• Learning to Compose Skills
• Floyd-Warshall Reinforcement Learning: Learning from Past Experiences to Reach New Goals
• Data-efficient Hindsight Off-policy Option Learning

Cited by in corpus (5)

• Structure in Deep Reinforcement Learning: A Survey and Open Problems
• Umbrella Reinforcement Learning -- computationally efficient tool for hard non-linear problems
• A Simple Approach to Continual Learning by Transferring Skill Parameters
• DAIR: Disentangled Attention Intrinsic Regularization for Safe and Efficient Bimanual Manipulation
• Learning to Design and Construct Bridge without Blueprint