Adversarial Skill Chaining for Long-Horizon Robot Manipulation via Terminal State Regularization

Abstract

Skill chaining is a promising approach for synthesizing complex behaviors by sequentially combining previously learned skills. Yet, a naive composition of skills fails when a policy encounters a starting state never seen during its training. For successful skill chaining, prior approaches attempt to widen the policy's starting state distribution. However, these approaches require larger state distributions to be covered as more policies are sequenced, and thus are limited to short skill sequences. In this paper, we propose to chain multiple policies without excessively large initial state distributions by regularizing the terminal state distributions in an adversarial learning framework. We evaluate our approach on two complex long-horizon manipulation tasks of furniture assembly. Our results have shown that our method establishes the first model-free reinforcement learning algorithm to solve these tasks; whereas prior skill chaining approaches fail. The code and videos are available at https://clvrai.com/skill-chaining

arXiv commentPublished at the Conference on Robot Learning (CoRL) 2021

References in corpus (15)

• Continuous control with deep reinforcement learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
• AMP: Adversarial Motion Priors for Stylized Physics-Based Character Control
• Generative Adversarial Imitation Learning
• Learning by Playing - Solving Sparse Reward Tasks from Scratch
• Modular Multitask Reinforcement Learning with Policy Sketches
• Meta Learning Shared Hierarchies
• Reinforcement and Imitation Learning for Diverse Visuomotor Skills
• Learning Robust Rewards with Adversarial Inverse Reinforcement Learning
• Learning Multi-Level Hierarchies with Hindsight
• Hierarchical visuomotor control of humanoids
• Task-Relevant Adversarial Imitation Learning
• Accelerating Reinforcement Learning with Learned Skill Priors
• Learning Task Decomposition with Ordered Memory Policy Network