ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrations

Abstract

Object manipulation from 3D visual inputs poses many challenges on building generalizable perception and policy models. However, 3D assets in existing benchmarks mostly lack the diversity of 3D shapes that align with real-world intra-class complexity in topology and geometry. Here we propose SAPIEN Manipulation Skill Benchmark (ManiSkill) to benchmark manipulation skills over diverse objects in a full-physics simulator. 3D assets in ManiSkill include large intra-class topological and geometric variations. Tasks are carefully chosen to cover distinct types of manipulation challenges. Latest progress in 3D vision also makes us believe that we should customize the benchmark so that the challenge is inviting to researchers working on 3D deep learning. To this end, we simulate a moving panoramic camera that returns ego-centric point clouds or RGB-D images. In addition, we would like ManiSkill to serve a broad set of researchers interested in manipulation research. Besides supporting the learning of policies from interactions, we also support learning-from-demonstrations (LfD) methods, by providing a large number of high-quality demonstrations (~36,000 successful trajectories, ~1.5M point cloud/RGB-D frames in total). We provide baselines using 3D deep learning and LfD algorithms. All code of our benchmark (simulator, environment, SDK, and baselines) is open-sourced, and a challenge facing interdisciplinary researchers will be held based on the benchmark.

arXiv commentNeurIPS 2021 Track on Datasets and Benchmarks; code: https://github.com/haosulab/ManiSkill

References in corpus (29)

• Soft Actor-Critic Algorithms and Applications
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• Solving Rubik's Cube with a Robot Hand
• Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
• Hindsight Experience Replay
• Submanifold Sparse Convolutional Networks
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning
• AI2-THOR: An Interactive 3D Environment for Visual AI
• Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics
• Building Generalizable Agents with a Realistic and Rich 3D Environment
• Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World
• dm_control: Software and Tasks for Continuous Control
• Leveraging Procedural Generation to Benchmark Reinforcement Learning
• robosuite: A Modular Simulation Framework and Benchmark for Robot Learning
• A Minimalist Approach to Offline Reinforcement Learning
• ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation
• Learning Invariant Feature Spaces to Transfer Skills with Reinforcement Learning
• Rearrangement: A Challenge for Embodied AI
• Articulated Object Interaction in Unknown Scenes with Whole-Body Mobile Manipulation
• Scaling data-driven robotics with reward sketching and batch reinforcement learning
• Transporter Networks: Rearranging the Visual World for Robotic Manipulation
• Learning Dexterous Manipulation Policies from Experience and Imitation
• ReLMoGen: Leveraging Motion Generation in Reinforcement Learning for Mobile Manipulation
• Asymmetric self-play for automatic goal discovery in robotic manipulation
• DoorGym: A Scalable Door Opening Environment And Baseline Agent
• RoboNet: Large-Scale Multi-Robot Learning
• Learning Sparse Rewarded Tasks from Sub-Optimal Demonstrations
• HRL4IN: Hierarchical Reinforcement Learning for Interactive Navigation with Mobile Manipulators