Open X-Embodiment: Robotic Learning Datasets and RT-X Models

Abstract

Large, high-capacity models trained on diverse datasets have shown remarkable successes on efficiently tackling downstream applications. In domains from NLP to Computer Vision, this has led to a consolidation of pretrained models, with general pretrained backbones serving as a starting point for many applications. Can such a consolidation happen in robotics? Conventionally, robotic learning methods train a separate model for every application, every robot, and even every environment. Can we instead train generalist X-robot policy that can be adapted efficiently to new robots, tasks, and environments? In this paper, we provide datasets in standardized data formats and models to make it possible to explore this possibility in the context of robotic manipulation, alongside experimental results that provide an example of effective X-robot policies. We assemble a dataset from 22 different robots collected through a collaboration between 21 institutions, demonstrating 527 skills (160266 tasks). We show that a high-capacity model trained on this data, which we call RT-X, exhibits positive transfer and improves the capabilities of multiple robots by leveraging experience from other platforms. More details can be found on the project website https://robotics-transformer-x.github.io.

Project website: https://robotics-transformer-x.github.io

Cited by in corpus (11)

• Challenges for Monocular 6D Object Pose Estimation in Robotics
• Knowledge Transfer for Cross-Domain Reinforcement Learning: A Systematic Review
• A roadmap for AI in robotics
• Robotic Applications of Pre-Trained Vision-Language Models to Various Recognition Behaviors
• Towards a Wireless Physical-Layer Foundation Model: Challenges and Strategies
• Transforming Agency. On the mode of existence of Large Language Models
• A Survey of Behavior Learning Applications in Robotics -- State of the Art and Perspectives
• Click to Grasp: Zero-Shot Precise Manipulation via Visual Diffusion Descriptors
• Correspondence learning between morphologically different robots via task demonstrations
• Interactive Learning of Physical Object Properties Through Robot Manipulation and Database of Object Measurements
• Single-grasp deformable object discrimination: the effect of gripper morphology, sensing modalities, and action parameters