Orbit: A Unified Simulation Framework for Interactive Robot Learning Environments

Abstract

We present Orbit, a unified and modular framework for robot learning powered by NVIDIA Isaac Sim. It offers a modular design to easily and efficiently create robotic environments with photo-realistic scenes and high-fidelity rigid and deformable body simulation. With Orbit, we provide a suite of benchmark tasks of varying difficulty -- from single-stage cabinet opening and cloth folding to multi-stage tasks such as room reorganization. To support working with diverse observations and action spaces, we include fixed-arm and mobile manipulators with different physically-based sensors and motion generators. Orbit allows training reinforcement learning policies and collecting large demonstration datasets from hand-crafted or expert solutions in a matter of minutes by leveraging GPU-based parallelization. In summary, we offer an open-sourced framework that readily comes with 16 robotic platforms, 4 sensor modalities, 10 motion generators, more than 20 benchmark tasks, and wrappers to 4 learning libraries. With this framework, we aim to support various research areas, including representation learning, reinforcement learning, imitation learning, and task and motion planning. We hope it helps establish interdisciplinary collaborations in these communities, and its modularity makes it easily extensible for more tasks and applications in the future.

Project website: https://isaac-orbit.github.io/

References in corpus (12)

• Learning agile and dynamic motor skills for legged robots
• The Replica Dataset: A Digital Replica of Indoor Spaces
• Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
• robosuite: A Modular Simulation Framework and Benchmark for Robot Learning
• ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation
• Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning
• Articulated Object Interaction in Unknown Scenes with Whole-Body Mobile Manipulation
• What Matters in Learning from Offline Human Demonstrations for Robot Manipulation
• SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulation
• iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Tasks
• Sim-to-Real for Soft Robots using Differentiable FEM: Recipes for Meshing, Damping, and Actuation
• ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skills

Cited by in corpus (17)

• Vision-Language-Action Models for Robotics: A Review Towards Real-World Applications
• Pegasus Simulator: An Isaac Sim Framework for Multiple Aerial Vehicles Simulation
• Deep Reinforcement Learning for Bipedal Locomotion: A Brief Survey
• Learn to Teach: Sample-Efficient Privileged Learning for Humanoid Locomotion over Diverse Terrains
• Dynamic object goal pushing with mobile manipulators through model-free constrained reinforcement learning
• MATTERIX: toward a digital twin for robotics-assisted chemistry laboratory automation
• RENet: Fault-Tolerant Motion Control for Quadruped Robots via Redundant Estimator Networks under Visual Collapse
• Isaac Sim-to-Real: Reinforcement Learning based Locomotion for Quadrupeds
• End-to-End Crop Row Navigation via LiDAR-Based Deep Reinforcement Learning
• Autonomous Legged Mobile Manipulation for Lunar Surface Operations via Constrained Reinforcement Learning
• Learning Terrain-Specialized Policies for Adaptive Locomotion in Challenging Environments
• SHIELD: Safety on Humanoids via CBFs In Expectation on Learned Dynamics
• ASBI: Leveraging Informative Real-World Data for Active Black-Box Simulator Tuning
• Realistic Data Generation for 6D Pose Estimation of Surgical Instruments
• Autonomous Planning In-space Assembly Reinforcement-learning free-flYer (APIARY) International Space Station Astrobee Testing
• Semantic Enrichment of CAD-Based Industrial Environments via Scene Graphs for Simulation and Reasoning
• SPLIT: Separating Physical-Contact via Latent Arithmetic in Image-Based Tactile Sensors