Solving Rubik's Cube with a Robot Hand

Abstract

We demonstrate that models trained only in simulation can be used to solve a manipulation problem of unprecedented complexity on a real robot. This is made possible by two key components: a novel algorithm, which we call automatic domain randomization (ADR) and a robot platform built for machine learning. ADR automatically generates a distribution over randomized environments of ever-increasing difficulty. Control policies and vision state estimators trained with ADR exhibit vastly improved sim2real transfer. For control policies, memory-augmented models trained on an ADR-generated distribution of environments show clear signs of emergent meta-learning at test time. The combination of ADR with our custom robot platform allows us to solve a Rubik's cube with a humanoid robot hand, which involves both control and state estimation problems. Videos summarizing our results are available: https://openai.com/blog/solving-rubiks-cube/

References in corpus (32)

• Soft Actor-Critic Algorithms and Applications
• Learning agile and dynamic motor skills for legged robots
• Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
• Robots that can adapt like animals
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• AutoAugment: Learning Augmentation Policies from Data
• Robust Adversarial Reinforcement Learning
• Learning to reinforcement learn
• Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics
• Exploration by Random Network Distillation
• Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World
• Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research
• Transfer from Simulation to Real World through Learning Deep Inverse Dynamics Model
• SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning
• Reinforcement and Imitation Learning for Diverse Visuomotor Skills
• Sim-to-Real Robot Learning from Pixels with Progressive Nets
• Learning Invariant Feature Spaces to Transfer Skills with Reinforcement Learning
• Deep Dynamics Models for Learning Dexterous Manipulation
• Meta reinforcement learning as task inference
• Unsupervised Meta-Learning for Reinforcement Learning
• AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence
• Learning Dexterous Manipulation Policies from Experience and Imitation
• Distilling Policy Distillation
• Reinforcement Learning for Pivoting Task
• Active Domain Randomization
• Concurrent Meta Reinforcement Learning
• POWERPLAY: Training an Increasingly General Problem Solver by Continually Searching for the Simplest Still Unsolvable Problem
• ORRB -- OpenAI Remote Rendering Backend
• First Experiments with PowerPlay
• Task-Oriented Hand Motion Retargeting for Dexterous Manipulation Imitation
• A Model-based Approach for Sample-efficient Multi-task Reinforcement Learning
• Learning to Solve a Rubik's Cube with a Dexterous Hand

Cited by in corpus (210)

• On the Opportunities and Risks of Foundation Models
• Shortcut Learning in Deep Neural Networks
• Dota 2 with Large Scale Deep Reinforcement Learning
• Multi-Task Learning with Deep Neural Networks: A Survey
• The Unreasonable Effectiveness of Deep Learning in Artificial Intelligence
• The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence
• WILDS: A Benchmark of in-the-Wild Distribution Shifts
• Reinforcement Learning with Augmented Data
• A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
• TACTO: A Fast, Flexible, and Open-source Simulator for High-Resolution Vision-based Tactile Sensors
• IG-RL: Inductive Graph Reinforcement Learning for Massive-Scale Traffic Signal Control
• What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study
• CLIPort: What and Where Pathways for Robotic Manipulation
• Balsa: Learning a Query Optimizer Without Expert Demonstrations
• Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
• Learning to Walk in the Real World with Minimal Human Effort
• Unity Perception: Generate Synthetic Data for Computer Vision
• Surprisingly Robust In-Hand Manipulation: An Empirical Study
• Measuring and modeling the motor system with machine learning
• Queueing Network Controls via Deep Reinforcement Learning
• Defining the scope of AI regulations
• One for Many: Transfer Learning for Building HVAC Control
• Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning
• Open-Ended Learning Leads to Generally Capable Agents
• Phasic Policy Gradient
• Transferring Dexterous Manipulation from GPU Simulation to a Remote Real-World TriFinger
• Learning Continuous Grasping Function with a Dexterous Hand from Human Demonstrations
• The Free Energy Principle for Perception and Action: A Deep Learning Perspective
• Generative Language Modeling for Automated Theorem Proving
• Never Stop Learning: The Effectiveness of Fine-Tuning in Robotic Reinforcement Learning
• Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations
• Reinforcement learning of rare diffusive dynamics
• MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale
• Automatic Curriculum Learning through Value Disagreement
• Perspectives on Sim2Real Transfer for Robotics: A Summary of the R:SS 2020 Workshop
• CausalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learning
• Reinforcement Learning with General Value Function Approximation: Provably Efficient Approach via Bounded Eluder Dimension
• ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrations
• Information Theoretic Regret Bounds for Online Nonlinear Control
• Online and Offline Reinforcement Learning by Planning with a Learned Model
• Scaling Laws for Transfer
• One Policy to Control Them All: Shared Modular Policies for Agent-Agnostic Control
• Towards Resilient Artificial Intelligence: Survey and Research Issues
• Learning to Fly via Deep Model-Based Reinforcement Learning
• Generalization in Dexterous Manipulation via Geometry-Aware Multi-Task Learning
• Network Environment Design for Autonomous Cyberdefense
• Forward and inverse reinforcement learning sharing network weights and hyperparameters
• Sample-efficient Reinforcement Learning Representation Learning with Curiosity Contrastive Forward Dynamics Model
• Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulation
• Asymmetric self-play for automatic goal discovery in robotic manipulation
• Benchmarking Robot Manipulation with the Rubik's Cube
• On the Role of the Action Space in Robot Manipulation Learning and Sim-to-Real Transfer
• Whole-Body Control of a Mobile Manipulator using End-to-End Reinforcement Learning
• Learning Connectivity-Maximizing Network Configurations
• Robust Policies via Mid-Level Visual Representations: An Experimental Study in Manipulation and Navigation
• Robotic Table Tennis: A Case Study into a High Speed Learning System
• Continual Robot Learning using Self-Supervised Task Inference
• Lipschitzness Is All You Need To Tame Off-policy Generative Adversarial Imitation Learning
• Synthesize Dexterous Nonprehensile Pregrasp for Ungraspable Objects
• Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning
• Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills
• Single-Timescale Actor-Critic Provably Finds Globally Optimal Policy
• Automating Turbulence Modeling by Multi-Agent Reinforcement Learning
• An Imitation from Observation Approach to Transfer Learning with Dynamics Mismatch
• Analysis of Randomization Effects on Sim2Real Transfer in Reinforcement Learning for Robotic Manipulation Tasks
• Variational Automatic Curriculum Learning for Sparse-Reward Cooperative Multi-Agent Problems
• Curriculum Learning for Safe Mapless Navigation
• Nonstationary Reinforcement Learning with Linear Function Approximation
• Revisiting Design Choices in Proximal Policy Optimization
• Estimator-Coupled Reinforcement Learning for Robust Purely Tactile In-Hand Manipulation
• Learning Object Manipulation Skills via Approximate State Estimation from Real Videos
• D2RL: Deep Dense Architectures in Reinforcement Learning
• The Distracting Control Suite -- A Challenging Benchmark for Reinforcement Learning from Pixels
• PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training
• Invariant Policy Optimization: Towards Stronger Generalization in Reinforcement Learning
• Enforcing robust control guarantees within neural network policies
• Robust Predictable Control
• Learning Affordance Landscapes for Interaction Exploration in 3D Environments
• CARLA Real Traffic Scenarios -- novel training ground and benchmark for autonomous driving
• Hierarchical Quality-Diversity for Online Damage Recovery
• Behavior From the Void: Unsupervised Active Pre-Training
• SimGAN: Hybrid Simulator Identification for Domain Adaptation via Adversarial Reinforcement Learning
• Systematic Generalisation through Task Temporal Logic and Deep Reinforcement Learning
• Evaluating Agents without Rewards
• Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observability
• Robust Policy Gradient against Strong Data Corruption
• Traversing the Reality Gap via Simulator Tuning
• Lyapunov Barrier Policy Optimization
• Affordance-based Reinforcement Learning for Urban Driving
• Multi-task curriculum learning in a complex, visual, hard-exploration domain: Minecraft
• Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL
• ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learning
• Augmented World Models Facilitate Zero-Shot Dynamics Generalization From a Single Offline Environment
• Evaluating the Robustness of Collaborative Agents
• On the Benefit of Optimal Transport for Curriculum Reinforcement Learning
• Deception in Social Learning: A Multi-Agent Reinforcement Learning Perspective
• Probabilistic Active Meta-Learning
• Curriculum in Gradient-Based Meta-Reinforcement Learning
• Measuring Visual Generalization in Continuous Control from Pixels
• Learning compositional models of robot skills for task and motion planning
• Circus ANYmal: A Quadruped Learning Dexterous Manipulation with Its Limbs
• Skill Preferences: Learning to Extract and Execute Robotic Skills from Human Feedback
• Bimanual Grasp Synthesis for Dexterous Robot Hands
• How to Make Deep RL Work in Practice
• Deep Reinforcement Learning with Embedded LQR Controllers
• RetinaGAN: An Object-aware Approach to Sim-to-Real Transfer
• The MAGICAL Benchmark for Robust Imitation
• Interactive Imitation Learning for Dexterous Robotic Manipulation: Challenges and Perspectives -- A Survey
• Co-GAIL: Learning Diverse Strategies for Human-Robot Collaboration
• Interpretable Model-based Hierarchical Reinforcement Learning using Inductive Logic Programming
• Attention-Privileged Reinforcement Learning
• Physically Embedded Planning Problems: New Challenges for Reinforcement Learning
• Policy Learning Using Weak Supervision
• Dynamics Randomization Revisited:A Case Study for Quadrupedal Locomotion
• MUSBO: Model-based Uncertainty Regularized and Sample Efficient Batch Optimization for Deployment Constrained Reinforcement Learning
• Learning Barrier Certificates: Towards Safe Reinforcement Learning with Zero Training-time Violations
• Relevance-guided Unsupervised Discovery of Abilities with Quality-Diversity Algorithms
• Network Defense is Not a Game
• Meta Automatic Curriculum Learning
• Mutual Information Based Knowledge Transfer Under State-Action Dimension Mismatch
• A Multi-Agent Approach for Adaptive Finger Cooperation in Learning-based In-Hand Manipulation
• Resolving Spurious Correlations in Causal Models of Environments via Interventions
• Interferobot: aligning an optical interferometer by a reinforcement learning agent
• Solving Challenging Dexterous Manipulation Tasks With Trajectory Optimisation and Reinforcement Learning
• Symbolic Behaviour in Artificial Intelligence
• CROP: Certifying Robust Policies for Reinforcement Learning through Functional Smoothing
• Meta-Reinforcement Learning in Broad and Non-Parametric Environments
• Ensuring Monotonic Policy Improvement in Entropy-regularized Value-based Reinforcement Learning
• Auto-Tuned Sim-to-Real Transfer
• Disentangled Planning and Control in Vision Based Robotics via Reward Machines
• Few Shot System Identification for Reinforcement Learning
• Benchmarking Structured Policies and Policy Optimization for Real-World Dexterous Object Manipulation
• Independent Natural Policy Gradient Always Converges in Markov Potential Games
• Machine Learning for Mechanical Ventilation Control
• Follow the Gradient: Crossing the Reality Gap using Differentiable Physics (RealityGrad)
• Learning Active Task-Oriented Exploration Policies for Bridging the Sim-to-Real Gap
• Learning more skills through optimistic exploration
• Learning of Balance Controller Considering Changes in Body State for Musculoskeletal Humanoids
• Mungojerrie: Reinforcement Learning of Linear-Time Objectives
• Adaptive Procedural Task Generation for Hard-Exploration Problems
• Regression as Classification: Influence of Task Formulation on Neural Network Features
• Guided Imitation of Task and Motion Planning
• Distributed Reinforcement Learning of Targeted Grasping with Active Vision for Mobile Manipulators
• Scalable sim-to-real transfer of soft robot designs
• Grasping in the Dark: Zero-Shot Object Grasping Using Tactile Feedback
• Machine Learning for Robotic Manipulation
• REALab: An Embedded Perspective on Tampering
• Learning to Play by Imitating Humans
• Co-Training an Observer and an Evading Target
• SeRO: Self-Supervised Reinforcement Learning for Recovery from Out-of-Distribution Situations
• A Simple Approach to Continual Learning by Transferring Skill Parameters
• Domain Adaptation of Networks for Camera Pose Estimation: Learning Camera Pose Estimation Without Pose Labels
• Online Sub-Sampling for Reinforcement Learning with General Function Approximation
• Pre-training of Deep RL Agents for Improved Learning under Domain Randomization
• Fault-Aware Robust Control via Adversarial Reinforcement Learning
• Towards Complex and Continuous Manipulation: A Gesture Based Anthropomorphic Robotic Hand Design
• Efficient and Interpretable Robot Manipulation with Graph Neural Networks
• Unlocking Pixels for Reinforcement Learning via Implicit Attention
• LISPR: An Options Framework for Policy Reuse with Reinforcement Learning
• Evaluating the Safety of Deep Reinforcement Learning Models using Semi-Formal Verification
• Follow the Object: Curriculum Learning for Manipulation Tasks with Imagined Goals
• A coevolutionary approach to deep multi-agent reinforcement learning
• Learning Compositional Neural Programs for Continuous Control
• Implicitly Regularized RL with Implicit Q-Values
• Reinforcement Learning with Time-dependent Goals for Robotic Musicians
• Global Convergence of the ODE Limit for Online Actor-Critic Algorithms in Reinforcement Learning
• Protective Policy Transfer
• Replay-Guided Adversarial Environment Design
• "What, not how": Solving an under-actuated insertion task from scratch
• Reinforcement Learning for Vision-based Object Manipulation with Non-parametric Policy and Action Primitives
• DROID: Minimizing the Reality Gap using Single-Shot Human Demonstration
• Questions to Guide the Future of Artificial Intelligence Research
• Discovering Generalizable Skills via Automated Generation of Diverse Tasks
• MMD-MIX: Value Function Factorisation with Maximum Mean Discrepancy for Cooperative Multi-Agent Reinforcement Learning
• OTTR: Off-Road Trajectory Tracking using Reinforcement Learning
• Do Robots Really Need Anthropomorphic Hands? A Comparison of Human and Robotic Hands
• Learning Fast and Precise Pixel-to-Torque Control
• Wish you were here: Hindsight Goal Selection for long-horizon dexterous manipulation
• Self-Adapting Recurrent Models for Object Pushing from Learning in Simulation
• Synthetic training data generation for deep learning based quality inspection
• Deep Reinforcement Learning with Linear Quadratic Regulator Regions
• Scene Graph Modification Based on Natural Language Commands
• SyDog: A Synthetic Dog Dataset for Improved 2D Pose Estimation
• Maximum Entropy Model-based Reinforcement Learning
• Leveraging Sequentiality in Reinforcement Learning from a Single Demonstration
• Trying AGAIN instead of Trying Longer: Prior Learning for Automatic Curriculum Learning
• Predicting Sim-to-Real Transfer with Probabilistic Dynamics Models
• Curriculum-based Sensing Reduction in Simulation to Real-World Transfer for In-hand Manipulation
• Regret Bounds for Adaptive Nonlinear Control
• Improved Reinforcement Learning Coordinated Control of a Mobile Manipulator using Joint Clamping
• DeepClaw: A Robotic Hardware Benchmarking Platform for Learning Object Manipulation
• Unsupervised Skill-Discovery and Skill-Learning in Minecraft
• Open Loop In Natura Economic Planning
• IV-Posterior: Inverse Value Estimation for Interpretable Policy Certificates
• Solving the Real Robot Challenge using Deep Reinforcement Learning
• Generalizing HVAC Control With Domain Randomized Reinforcement Learning
• Learning to Design and Construct Bridge without Blueprint
• RL agents Implicitly Learning Human Preferences
• Contrastive Active Inference
• NEARL: Non-Explicit Action Reinforcement Learning for Robotic Control
• Learning Provably Robust Motion Planners Using Funnel Libraries
• A Broad-persistent Advising Approach for Deep Interactive Reinforcement Learning in Robotic Environments
• Adaptation of Quadruped Robot Locomotion with Meta-Learning
• Playing optical tweezers with deep reinforcement learning: in virtual, physical and augmented environments
• Teachable Reinforcement Learning via Advice Distillation
• Towards Learning to Play Piano with Dexterous Hands and Touch
• Learning a Skill-sequence-dependent Policy for Long-horizon Manipulation Tasks
• Learning to Manipulate Amorphous Materials
• Deep Reinforcement Learning for Tactile Robotics: Learning to Type on a Braille Keyboard
• Learning robust driving policies without online exploration