Learning a Skill-sequence-dependent Policy for Long-horizon Manipulation Tasks

Abstract

In recent years, the robotics community has made substantial progress in robotic manipulation using deep reinforcement learning (RL). Effectively learning of long-horizon tasks remains a challenging topic. Typical RL-based methods approximate long-horizon tasks as Markov decision processes and only consider current observation (images or other sensor information) as input state. However, such approximation ignores the fact that skill-sequence also plays a crucial role in long-horizon tasks. In this paper, we take both the observation and skill sequences into account and propose a skill-sequence-dependent hierarchical policy for solving a typical long-horizon task. The proposed policy consists of a high-level skill policy (utilizing skill sequences) and a low-level parameter policy (responding to observation) with corresponding training methods, which makes the learning much more sample-efficient. Experiments in simulation demonstrate that our approach successfully solves a long-horizon task and is significantly faster than Proximal Policy Optimization (PPO) and the task schema methods.

References in corpus (10)

• Emergence of Locomotion Behaviours in Rich Environments
• Solving Rubik's Cube with a Robot Hand
• Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
• robosuite: A Modular Simulation Framework and Benchmark for Robot Learning
• Learning Predictive Representations for Deformable Objects Using Contrastive Estimation
• ReLMoGen: Leveraging Motion Generation in Reinforcement Learning for Mobile Manipulation
• HIRL: Hierarchical Inverse Reinforcement Learning for Long-Horizon Tasks with Delayed Rewards
• Hierarchical Foresight: Self-Supervised Learning of Long-Horizon Tasks via Visual Subgoal Generation
• Modeling Long-horizon Tasks as Sequential Interaction Landscapes
• Hierarchical Reinforcement Learning for Quadruped Locomotion