Learning Dexterous In-Hand Manipulation

Abstract

We use reinforcement learning (RL) to learn dexterous in-hand manipulation policies which can perform vision-based object reorientation on a physical Shadow Dexterous Hand. The training is performed in a simulated environment in which we randomize many of the physical properties of the system like friction coefficients and an object's appearance. Our policies transfer to the physical robot despite being trained entirely in simulation. Our method does not rely on any human demonstrations, but many behaviors found in human manipulation emerge naturally, including finger gaiting, multi-finger coordination, and the controlled use of gravity. Our results were obtained using the same distributed RL system that was used to train OpenAI Five. We also include a video of our results: https://youtu.be/jwSbzNHGflM

Making OpenAI the first author. We wish this paper to be cited as "Learning Dexterous In-Hand Manipulation" by OpenAI et al. We are replicating the approach from the physics community: arXiv:1812.06489

References in corpus (14)

• Adam: A Method for Stochastic Optimization
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• Robust Adversarial Reinforcement Learning
• Distributed Distributional Deterministic Policy Gradients
• Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics
• Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World
• Transfer from Simulation to Real World through Learning Deep Inverse Dynamics Model
• Reinforcement and Imitation Learning for Diverse Visuomotor Skills
• Sim-to-Real Robot Learning from Pixels with Progressive Nets
• Learning Invariant Feature Spaces to Transfer Skills with Reinforcement Learning
• Learning Dexterous Manipulation Policies from Experience and Imitation
• Reinforcement Learning for Pivoting Task

Cited by in corpus (109)

• A Survey of Deep Learning Techniques for Autonomous Driving
• DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
• Hamiltonian Neural Networks
• Emergent Tool Use From Multi-Agent Autocurricula
• Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
• Agent57: Outperforming the Atari Human Benchmark
• Go-Explore: a New Approach for Hard-Exploration Problems
• Ablation Studies in Artificial Neural Networks
• DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections
• What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study
• Implicit Generation and Generalization in Energy-Based Models
• Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
• Surprisingly Robust In-Hand Manipulation: An Empirical Study
• Deep Dynamics Models for Learning Dexterous Manipulation
• Assessing Transferability from Simulation to Reality for Reinforcement Learning
• An End-to-End Differentiable Framework for Contact-Aware Robot Design
• Defining the scope of AI regulations
• PyRobot: An Open-source Robotics Framework for Research and Benchmarking
• Multi-Fingered In-Hand Manipulation with Various Object Properties Using Graph Convolutional Networks and Distributed Tactile Sensors
• Benchmarking In-Hand Manipulation
• An empirical investigation of the challenges of real-world reinforcement learning
• Generative Teaching Networks: Accelerating Neural Architecture Search by Learning to Generate Synthetic Training Data
• Robust, Occlusion-aware Pose Estimation for Objects Grasped by Adaptive Hands
• Scaling data-driven robotics with reward sketching and batch reinforcement learning
• Robust Reinforcement Learning for Continuous Control with Model Misspecification
• Deep Q-Network Based Decision Making for Autonomous Driving
• Learning Purely Tactile In-Hand Manipulation with a Torque-Controlled Hand
• COG: Connecting New Skills to Past Experience with Offline Reinforcement Learning
• Reinforcement Learning for Robotic Manipulation using Simulated Locomotion Demonstrations
• Making Deep Q-learning methods robust to time discretization
• Active Domain Randomization
• Dextrous Tactile In-Hand Manipulation Using a Modular Reinforcement Learning Architecture
• DeepRacer: Educational Autonomous Racing Platform for Experimentation with Sim2Real Reinforcement Learning
• ORL: Reinforcement Learning Benchmarks for Online Stochastic Optimization Problems
• Analysing Deep Reinforcement Learning Agents Trained with Domain Randomisation
• IPO: Interior-point Policy Optimization under Constraints
• REPLAB: A Reproducible Low-Cost Arm Benchmark Platform for Robotic Learning
• Self-driving scale car trained by Deep reinforcement learning
• Towards Practical Multi-Object Manipulation using Relational Reinforcement Learning
• VisuoSpatial Foresight for Multi-Step, Multi-Task Fabric Manipulation
• Learning to Manipulate Deformable Objects without Demonstrations
• Dexterous Manipulation with Deep Reinforcement Learning: Efficient, General, and Low-Cost
• GPU-Accelerated Robotic Simulation for Distributed Reinforcement Learning
• A hybrid learning method for system identification and optimal control
• AC-Teach: A Bayesian Actor-Critic Method for Policy Learning with an Ensemble of Suboptimal Teachers
• Dynamics-aware Embeddings
• Reconfigurable Design for Omni-adaptive Grasp Learning
• RoboNet: Large-Scale Multi-Robot Learning
• Optimization Methods for Interpretable Differentiable Decision Trees in Reinforcement Learning
• Motion Planner Augmented Reinforcement Learning for Robot Manipulation in Obstructed Environments
• Flight Controller Synthesis Via Deep Reinforcement Learning
• Learning Visual Robotic Control Efficiently with Contrastive Pre-training and Data Augmentation
• Quickly Inserting Pegs into Uncertain Holes using Multi-view Images and Deep Network Trained on Synthetic Data
• Behavior Priors for Efficient Reinforcement Learning
• Discretizing Continuous Action Space for On-Policy Optimization
• Dual-Arm In-Hand Manipulation and Regrasping Using Dexterous Manipulation Graphs
• Pose Estimation for Non-Cooperative Rendezvous Using Neural Networks
• Estimator-Coupled Reinforcement Learning for Robust Purely Tactile In-Hand Manipulation
• On Multi-Agent Learning in Team Sports Games
• Bayesian Optimization in Variational Latent Spaces with Dynamic Compression
• ROBEL: Robotics Benchmarks for Learning with Low-Cost Robots
• Soft Hindsight Experience Replay
• Diverse Auto-Curriculum is Critical for Successful Real-World Multiagent Learning Systems
• Integrating Behavior Cloning and Reinforcement Learning for Improved Performance in Dense and Sparse Reward Environments
• Sample Complexity of Reinforcement Learning using Linearly Combined Model Ensembles
• Hardware as Policy: Mechanical and Computational Co-Optimization using Deep Reinforcement Learning
• Smooth Exploration for Robotic Reinforcement Learning
• OffWorld Gym: open-access physical robotics environment for real-world reinforcement learning benchmark and research
• Autonomous Driving in the Lung using Deep Learning for Localization
• Vision-based Teleoperation of Shadow Dexterous Hand using End-to-End Deep Neural Network
• Robust Opponent Modeling via Adversarial Ensemble Reinforcement Learning in Asymmetric Imperfect-Information Games
• SUMMIT: A Simulator for Urban Driving in Massive Mixed Traffic
• Robust Visual Domain Randomization for Reinforcement Learning
• A Framework for Studying Reinforcement Learning and Sim-to-Real in Robot Soccer
• MARLIN: Soft Actor-Critic based Reinforcement Learning for Congestion Control in Real Networks
• Imagined Value Gradients: Model-Based Policy Optimization with Transferable Latent Dynamics Models
• Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learning
• Interferobot: aligning an optical interferometer by a reinforcement learning agent
• In-Hand Object Pose Tracking via Contact Feedback and GPU-Accelerated Robotic Simulation
• Blue River Controls: A toolkit for Reinforcement Learning Control Systems on Hardware
• Reaching, Grasping and Re-grasping: Learning Multimode Grasping Skills
• Model Based Planning with Energy Based Models
• Contextual Reinforcement Learning of Visuo-tactile Multi-fingered Grasping Policies
• Learning Cross-Domain Correspondence for Control with Dynamics Cycle-Consistency
• Robust Constrained Reinforcement Learning for Continuous Control with Model Misspecification
• Lyceum: An efficient and scalable ecosystem for robot learning
• Learning to Solve a Rubik's Cube with a Dexterous Hand
• Simulating Autonomous Driving in Massive Mixed Urban Traffic
• MANGA: Method Agnostic Neural-policy Generalization and Adaptation
• Competitive Experience Replay
• Reconnaissance and Planning algorithm for constrained MDP
• Explicitly Encouraging Low Fractional Dimensional Trajectories Via Reinforcement Learning
• Learning Pregrasp Manipulation of Objects from Ungraspable Poses
• Local Trajectory Stabilization for Dexterous Manipulation via Piecewise Affine Approximations
• Using reinforcement learning to minimize taxi idle times
• From semantics to execution: Integrating action planning with reinforcement learning for robotic causal problem-solving
• Learning Compositional Neural Programs for Continuous Control
• The utility of tactile force to autonomous learning of in-hand manipulation is task-dependent
• Autonomous Learning of Features for Control: Experiments with Embodied and Situated Agents
• Practical Robot Learning from Demonstrations using Deep End-to-End Training
• Sample Complexity of Estimating the Policy Gradient for Nearly Deterministic Dynamical Systems
• An Adaptive Framework for Reliable Trajectory Following in Changing-Contact Robot Manipulation Tasks
• CoachNet: An Adversarial Sampling Approach for Reinforcement Learning
• Sim-to-Real Transfer of Robot Learning with Variable Length Inputs
• Autonomous Functional Locomotion in a Tendon-Driven Limb via Limited Experience
• Direct Random Search for Fine Tuning of Deep Reinforcement Learning Policies
• Reinforcement Learning in Topology-based Representation for Human Body Movement with Whole Arm Manipulation
• Robby is Not a Robber (anymore): On the Use of Institutions for Learning Normative Behavior
• Adaptation of Quadruped Robot Locomotion with Meta-Learning