ReLMoGen: Leveraging Motion Generation in Reinforcement Learning for Mobile Manipulation

Abstract

Many Reinforcement Learning (RL) approaches use joint control signals (positions, velocities, torques) as action space for continuous control tasks. We propose to lift the action space to a higher level in the form of subgoals for a motion generator (a combination of motion planner and trajectory executor). We argue that, by lifting the action space and by leveraging sampling-based motion planners, we can efficiently use RL to solve complex, long-horizon tasks that could not be solved with existing RL methods in the original action space. We propose ReLMoGen -- a framework that combines a learned policy to predict subgoals and a motion generator to plan and execute the motion needed to reach these subgoals. To validate our method, we apply ReLMoGen to two types of tasks: 1) Interactive Navigation tasks, navigation problems where interactions with the environment are required to reach the destination, and 2) Mobile Manipulation tasks, manipulation tasks that require moving the robot base. These problems are challenging because they are usually long-horizon, hard to explore during training, and comprise alternating phases of navigation and interaction. Our method is benchmarked on a diverse set of seven robotics tasks in photo-realistic simulation environments. In all settings, ReLMoGen outperforms state-of-the-art Reinforcement Learning and Hierarchical Reinforcement Learning baselines. ReLMoGen also shows outstanding transferability between different motion generators at test time, indicating a great potential to transfer to real robots.

arXiv commentFirst two authors contributed equally. Access project website at http://svl.stanford.edu/projects/relmogen

References in corpus (21)

• Playing Atari with Deep Reinforcement Learning
• A Brief Survey of Deep Reinforcement Learning
• Soft Actor-Critic Algorithms and Applications
• Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• On Evaluation of Embodied Navigation Agents
• Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics
• FeUdal Networks for Hierarchical Reinforcement Learning
• Driving Policy Transfer via Modularity and Abstraction
• Deep Exploration via Randomized Value Functions
• Sim-to-Real: Learning Agile Locomotion For Quadruped Robots
• Learning and Transfer of Modulated Locomotor Controllers
• Extending the OpenAI Gym for robotics: a toolkit for reinforcement learning using ROS and Gazebo
• Spatial Action Maps for Mobile Manipulation
• Learning Multi-Level Hierarchies with Hindsight
• Near-Optimal Representation Learning for Hierarchical Reinforcement Learning
• Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?
• rlpyt: A Research Code Base for Deep Reinforcement Learning in PyTorch
• Better Exploration with Optimistic Actor-Critic
• Integrating Task-Motion Planning with Reinforcement Learning for Robust Decision Making in Mobile Robots
• Efficient Exploration in Constrained Environments with Goal-Oriented Reference Path

Cited by in corpus (12)

• Articulated Object Interaction in Unknown Scenes with Whole-Body Mobile Manipulation
• iGibson 1.0: a Simulation Environment for Interactive Tasks in Large Realistic Scenes
• N${}^2$M${}^2$: Learning Navigation for Arbitrary Mobile Manipulation Motions in Unseen and Dynamic Environments
• ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrations
• Motion Planner Augmented Reinforcement Learning for Robot Manipulation in Obstructed Environments
• Learning Pneumatic Non-Prehensile Manipulation with a Mobile Blower
• The ThreeDWorld Transport Challenge: A Visually Guided Task-and-Motion Planning Benchmark for Physically Realistic Embodied AI
• Benchmarking Structured Policies and Policy Optimization for Real-World Dexterous Object Manipulation
• Solving Challenging Control Problems Using Two-Staged Deep Reinforcement Learning
• Error-Aware Imitation Learning from Teleoperation Data for Mobile Manipulation
• Self-Imitation Learning by Planning
• Learning a Skill-sequence-dependent Policy for Long-horizon Manipulation Tasks