QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation

Abstract

In this paper, we study the problem of learning vision-based dynamic manipulation skills using a scalable reinforcement learning approach. We study this problem in the context of grasping, a longstanding challenge in robotic manipulation. In contrast to static learning behaviors that choose a grasp point and then execute the desired grasp, our method enables closed-loop vision-based control, whereby the robot continuously updates its grasp strategy based on the most recent observations to optimize long-horizon grasp success. To that end, we introduce QT-Opt, a scalable self-supervised vision-based reinforcement learning framework that can leverage over 580k real-world grasp attempts to train a deep neural network Q-function with over 1.2M parameters to perform closed-loop, real-world grasping that generalizes to 96% grasp success on unseen objects. Aside from attaining a very high success rate, our method exhibits behaviors that are quite distinct from more standard grasping systems: using only RGB vision-based perception from an over-the-shoulder camera, our method automatically learns regrasping strategies, probes objects to find the most effective grasps, learns to reposition objects and perform other non-prehensile pre-grasp manipulations, and responds dynamically to disturbances and perturbations.

arXiv commentCoRL 2018 camera ready. 23 pages, 14 figures

References in corpus (9)

• Continuous control with deep reinforcement learning
• Benchmarking Deep Reinforcement Learning for Continuous Control
• Distributed Prioritized Experience Replay
• Massively Parallel Methods for Deep Reinforcement Learning
• Input Convex Neural Networks
• Collective Robot Reinforcement Learning with Distributed Asynchronous Guided Policy Search
• Learning a visuomotor controller for real world robotic grasping using simulated depth images
• Accelerated Methods for Deep Reinforcement Learning
• Deep Predictive Policy Training using Reinforcement Learning

Cited by in corpus (211)

• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Solving Rubik's Cube with a Robot Hand
• Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning
• Learning Dexterous In-Hand Manipulation
• Reinforcement Learning with Augmented Data
• CURL: Contrastive Unsupervised Representations for Reinforcement Learning
• Learning Force Control for Contact-rich Manipulation Tasks with Rigid Position-controlled Robots
• Lyapunov-based Safe Policy Optimization for Continuous Control
• UniGrasp: Learning a Unified Model to Grasp with Multifingered Robotic Hands
• Mechanical Search: Multi-Step Retrieval of a Target Object Occluded by Clutter
• FACMAC: Factored Multi-Agent Centralised Policy Gradients
• Model-Based Meta-Reinforcement Learning for Flight with Suspended Payloads
• Grasp2Vec: Learning Object Representations from Self-Supervised Grasping
• A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms
• Learning to Walk in the Real World with Minimal Human Effort
• Hybrid Robotic Grasping with a Soft Multimodal Gripper and a Deep Multistage Learning Scheme
• ChainerRL: A Deep Reinforcement Learning Library
• Dynamics-Aware Unsupervised Discovery of Skills
• Decoupling Representation Learning from Reinforcement Learning
• Self-Supervised Policy Adaptation during Deployment
• Towards Characterizing Divergence in Deep Q-Learning
• Neurosymbolic Reinforcement Learning and Planning: A Survey
• COMBO: Conservative Offline Model-Based Policy Optimization
• An empirical investigation of the challenges of real-world reinforcement learning
• Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning
• Adaptive Optics control using Model-Based Reinforcement Learning
• SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learning
• Scaling data-driven robotics with reward sketching and batch reinforcement learning
• Visual Foresight Trees for Object Retrieval from Clutter with Nonprehensile Rearrangement
• Controlling the Solo12 Quadruped Robot with Deep Reinforcement Learning
• Transporter Networks: Rearranging the Visual World for Robotic Manipulation
• RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitation
• Robust Reinforcement Learning for Continuous Control with Model Misspecification
• ReLMoGen: Leveraging Motion Generation in Reinforcement Learning for Mobile Manipulation
• DGCM-Net: Dense Geometrical Correspondence Matching Network for Incremental Experience-based Robotic Grasping
• Multi-modal Transfer Learning for Grasping Transparent and Specular Objects
• MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale
• DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correction
• Entity Abstraction in Visual Model-Based Reinforcement Learning
• Q-Learning in enormous action spaces via amortized approximate maximization
• Uncovering Instabilities in Variational-Quantum Deep Q-Networks
• Data Efficient Reinforcement Learning for Legged Robots
• ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrations
• EpidemiOptim: A Toolbox for the Optimization of Control Policies in Epidemiological Models
• The Ingredients of Real-World Robotic Reinforcement Learning
• Learning to Fly via Deep Model-Based Reinforcement Learning
• Off-Policy Evaluation via Off-Policy Classification
• Interpretable End-to-end Urban Autonomous Driving with Latent Deep Reinforcement Learning
• Benchmarks for Deep Off-Policy Evaluation
• Learning Latent Plans from Play
• Sample-efficient Reinforcement Learning Representation Learning with Curiosity Contrastive Forward Dynamics Model
• REPLAB: A Reproducible Low-Cost Arm Benchmark Platform for Robotic Learning
• Optimal Stroke Learning with Policy Gradient Approach for Robotic Table Tennis
• Recovery RL: Safe Reinforcement Learning with Learned Recovery Zones
• Generalizable Episodic Memory for Deep Reinforcement Learning
• Learning to Seek: Autonomous Source Seeking with Deep Reinforcement Learning Onboard a Nano Drone Microcontroller
• Landmark-Guided Subgoal Generation in Hierarchical Reinforcement Learning
• Learning to Grasp on the Moon from 3D Octree Observations with Deep Reinforcement Learning
• Learning Robotic Manipulation through Visual Planning and Acting
• AC-Teach: A Bayesian Actor-Critic Method for Policy Learning with an Ensemble of Suboptimal Teachers
• Dynamics-aware Embeddings
• Hierarchical Foresight: Self-Supervised Learning of Long-Horizon Tasks via Visual Subgoal Generation
• Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning
• Generalization in Reinforcement Learning by Soft Data Augmentation
• Hindsight Goal Ranking on Replay Buffer for Sparse Reward Environment
• Learning Dense Visual Correspondences in Simulation to Smooth and Fold Real Fabrics
• Motion Planner Augmented Reinforcement Learning for Robot Manipulation in Obstructed Environments
• X-Ray: Mechanical Search for an Occluded Object by Minimizing Support of Learned Occupancy Distributions
• Learning Task-relevant Representations for Generalization via Characteristic Functions of Reward Sequence Distributions
• Provably Efficient Online Hyperparameter Optimization with Population-Based Bandits
• Learning Visual Robotic Control Efficiently with Contrastive Pre-training and Data Augmentation
• Goal-Auxiliary Actor-Critic for 6D Robotic Grasping with Point Clouds
• Grasping as Inference: Reactive Grasping in Heavily Cluttered Environment
• On the Estimation Bias in Double Q-Learning
• Time Reversal as Self-Supervision
• Modeling Long-horizon Tasks as Sequential Interaction Landscapes
• D2RL: Deep Dense Architectures in Reinforcement Learning
• CAQL: Continuous Action Q-Learning
• Deep Reinforcement Learning for Autonomous Internet of Things: Model, Applications and Challenges
• Ctrl-Z: Recovering from Instability in Reinforcement Learning
• Scaling Robot Supervision to Hundreds of Hours with RoboTurk: Robotic Manipulation Dataset through Human Reasoning and Dexterity
• Residual Reinforcement Learning from Demonstrations
• ROBEL: Robotics Benchmarks for Learning with Low-Cost Robots
• Automata Guided Reinforcement Learning With Demonstrations
• Soft Hindsight Experience Replay
• Learning to Rearrange Deformable Cables, Fabrics, and Bags with Goal-Conditioned Transporter Networks
• RL4ReAl: Reinforcement Learning for Register Allocation
• Emergent Real-World Robotic Skills via Unsupervised Off-Policy Reinforcement Learning
• MAT: Multi-Fingered Adaptive Tactile Grasping via Deep Reinforcement Learning
• Grasping in the Wild:Learning 6DoF Closed-Loop Grasping from Low-Cost Demonstrations
• Automating Reinforcement Learning with Example-based Resets
• Lyapunov Barrier Policy Optimization
• Causal Influence Detection for Improving Efficiency in Reinforcement Learning
• Maximum Mutation Reinforcement Learning for Scalable Control
• Efficient reinforcement learning control for continuum robots based on Inexplicit Prior Knowledge
• Grasp-type Recognition Leveraging Object Affordance
• OffWorld Gym: open-access physical robotics environment for real-world reinforcement learning benchmark and research
• PlayVirtual: Augmenting Cycle-Consistent Virtual Trajectories for Reinforcement Learning
• Measuring Visual Generalization in Continuous Control from Pixels
• Learning from Guided Play: Improving Exploration for Adversarial Imitation Learning with Simple Auxiliary Tasks
• Learning Dense Rewards for Contact-Rich Manipulation Tasks
• Sub-Goal Trees -- a Framework for Goal-Based Reinforcement Learning
• Which Mutual-Information Representation Learning Objectives are Sufficient for Control?
• Acceleration of Actor-Critic Deep Reinforcement Learning for Visual Grasping in Clutter by State Representation Learning Based on Disentanglement of a Raw Input Image
• How to Make Deep RL Work in Practice
• EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL
• Carl-Lead: Lidar-based End-to-End Autonomous Driving with Contrastive Deep Reinforcement Learning
• RetinaGAN: An Object-aware Approach to Sim-to-Real Transfer
• Learning Self-Correctable Policies and Value Functions from Demonstrations with Negative Sampling
• Ab Initio Particle-based Object Manipulation
• Lifelong Robotic Reinforcement Learning by Retaining Experiences
• Trajectory-wise Multiple Choice Learning for Dynamics Generalization in Reinforcement Learning
• PointNet++ Grasping: Learning An End-to-end Spatial Grasp Generation Algorithm from Sparse Point Clouds
• Improving Computational Efficiency in Visual Reinforcement Learning via Stored Embeddings
• Multi-Fingered Active Grasp Learning
• Synergies Between Affordance and Geometry: 6-DoF Grasp Detection via Implicit Representations
• GRIMGEP: Learning Progress for Robust Goal Sampling in Visual Deep Reinforcement Learning
• Balance Between Efficient and Effective Learning: Dense2Sparse Reward Shaping for Robot Manipulation with Environment Uncertainty
• Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learning
• PsiPhi-Learning: Reinforcement Learning with Demonstrations using Successor Features and Inverse Temporal Difference Learning
• GRAC: Self-Guided and Self-Regularized Actor-Critic
• Robotic Table Tennis with Model-Free Reinforcement Learning
• Data-efficient visuomotor policy training using reinforcement learning and generative models
• Interferobot: aligning an optical interferometer by a reinforcement learning agent
• Visual-Policy Learning through Multi-Camera View to Single-Camera View Knowledge Distillation for Robot Manipulation Tasks
• Disentangled Planning and Control in Vision Based Robotics via Reward Machines
• AW-Opt: Learning Robotic Skills with Imitation and Reinforcement at Scale
• Iterative Amortized Policy Optimization
• The Importance and the Limitations of Sim2Real for Robotic Manipulation in Precision Agriculture
• Optimization Algorithm for Feedback and Feedforward Policies towards Robot Control Robust to Sensing Failures
• Pixel-Attentive Policy Gradient for Multi-Fingered Grasping in Cluttered Scenes
• AutoSoC: Automating Algorithm-SOC Co-design for Aerial Robots
• Efficient learning of goal-oriented push-grasping synergy in clutter
• Active Perception and Representation for Robotic Manipulation
• Object-Agnostic Suction Grasp Affordance Detection in Dense Cluster Using Self-Supervised Learning.docx
• Learning Dynamics Models for Model Predictive Agents
• Continual Learning: Tackling Catastrophic Forgetting in Deep Neural Networks with Replay Processes
• Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximation
• Distributed Reinforcement Learning of Targeted Grasping with Active Vision for Mobile Manipulators
• Reconnaissance and Planning algorithm for constrained MDP
• AutoPilot: Automating SoC Design Space Exploration for SWaP Constrained Autonomous UAVs
• Language Conditioned Imitation Learning over Unstructured Data
• Learning a Generative Transition Model for Uncertainty-Aware Robotic Manipulation
• Competitive Experience Replay
• Continuous Transition: Improving Sample Efficiency for Continuous Control Problems via MixUp
• MetaGrasp: Data Efficient Grasping by Affordance Interpreter Network
• Variational Empowerment as Representation Learning for Goal-Based Reinforcement Learning
• Sim-To-Real Transfer for Miniature Autonomous Car Racing
• Towards Automatic Actor-Critic Solutions to Continuous Control
• Scenic4RL: Programmatic Modeling and Generation of Reinforcement Learning Environments
• Learning to Scaffold the Development of Robotic Manipulation Skills
• Self-supervised Visual Reinforcement Learning with Object-centric Representations
• Roofline Model for UAVs:A Bottleneck Analysis Tool for Designing Compute Systems for Autonomous Drones
• Lyapunov-stable neural-network control
• Forgetful Experience Replay in Hierarchical Reinforcement Learning from Demonstrations
• Learning to Play by Imitating Humans
• Self-Imitation Learning by Planning
• Distributed Deep Reinforcement Learning: An Overview
• DIPN: Deep Interaction Prediction Network with Application to Clutter Removal
• Boosting Offline Reinforcement Learning with Residual Generative Modeling
• Continuous Control with Action Quantization from Demonstrations
• Aligning an optical interferometer with beam divergence control and continuous action space
• Policy Information Capacity: Information-Theoretic Measure for Task Complexity in Deep Reinforcement Learning
• PerSim: Data-Efficient Offline Reinforcement Learning with Heterogeneous Agents via Personalized Simulators
• Unlocking Pixels for Reinforcement Learning via Implicit Attention
• Inner Attention Supported Adaptive Cooperation for Heterogeneous Multi Robots Teaming based on Multi-agent Reinforcement Learning
• A Deep Learning Approach to Grasping the Invisible
• Data-Driven Model Predictive Control for Food-Cutting
• Auditing Robot Learning for Safety and Compliance during Deployment
• Robo-PlaNet: Learning to Poke in a Day
• Koopman Q-learning: Offline Reinforcement Learning via Symmetries of Dynamics
• Goal-constrained Sparse Reinforcement Learning for End-to-End Driving
• Tool as Embodiment for Recursive Manipulation
• Learning Principle of Least Action with Reinforcement Learning
• Comparing Task Simplifications to Learn Closed-Loop Object Picking Using Deep Reinforcement Learning
• Coinbot: Intelligent Robotic Coin Bag Manipulation Using Deep Reinforcement Learning And Machine Teaching
• Importance Sampling based Exploration in Q Learning
• Off-policy Reinforcement Learning with Optimistic Exploration and Distribution Correction
• Recognizing Object Affordances to Support Scene Reasoning for Manipulation Tasks
• Lyapunov-Based Reinforcement Learning for Decentralized Multi-Agent Control
• Investigation of Factorized Optical Flows as Mid-Level Representations
• Learning Off-Policy with Online Planning
• Reinforcement Learning for Vision-based Object Manipulation with Non-parametric Policy and Action Primitives
• Hierarchical Neural Dynamic Policies
• Lyapunov-based uncertainty-aware safe reinforcement learning
• Training an Interactive Helper
• Understanding the World Through Action
• Accelerating Grasp Exploration by Leveraging Learned Priors
• A Geometric Perspective on Self-Supervised Policy Adaptation
• ACDER: Augmented Curiosity-Driven Experience Replay
• Learning to Manipulate Amorphous Materials
• LyRN (Lyapunov Reaching Network): A Real-Time Closed Loop approach from Monocular Vision
• Self-supervised Learning for Precise Pick-and-place without Object Model
• When Autonomous Systems Meet Accuracy and Transferability through AI: A Survey
• Transfer Reinforcement Learning across Homotopy Classes
• Decoupled Exploration and Exploitation Policies for Sample-Efficient Reinforcement Learning
• Sim-to-Real Transfer of Robot Learning with Variable Length Inputs
• Learning Robotic Manipulation Tasks via Task Progress based Gaussian Reward and Loss Adjusted Exploration
• Adaptive Variants of Optimal Feedback Policies
• InsertionNet -- A Scalable Solution for Insertion
• Active Offline Policy Selection
• Hierarchical Policies for Cluttered-Scene Grasping with Latent Plans
• Uncertainty Quantification and Exploration for Reinforcement Learning
• On-Policy Model Errors in Reinforcement Learning
• Rearranging the Environment to Maximize Energy with a Robotic Circuit Drawing
• Teachable Reinforcement Learning via Advice Distillation
• Few-Shot Image-to-Semantics Translation for Policy Transfer in Reinforcement Learning
• Grasp Planning for Flexible Production with Small Lot Sizes based on CAD models using GPIS and Bayesian Optimization
• Variational Policy Search using Sparse Gaussian Process Priors for Learning Multimodal Optimal Actions
• Adaptation of Quadruped Robot Locomotion with Meta-Learning