Reinforcement and Imitation Learning for Diverse Visuomotor Skills

Abstract

We propose a model-free deep reinforcement learning method that leverages a small amount of demonstration data to assist a reinforcement learning agent. We apply this approach to robotic manipulation tasks and train end-to-end visuomotor policies that map directly from RGB camera inputs to joint velocities. We demonstrate that our approach can solve a wide variety of visuomotor tasks, for which engineering a scripted controller would be laborious. In experiments, our reinforcement and imitation agent achieves significantly better performances than agents trained with reinforcement learning or imitation learning alone. We also illustrate that these policies, trained with large visual and dynamics variations, can achieve preliminary successes in zero-shot sim2real transfer. A brief visual description of this work can be viewed in https://youtu.be/EDl8SQUNjj0

arXiv comment13 pages, 6 figures, Published in RSS 2018

References in corpus (21)

• Adam: A Method for Stochastic Optimization
• End-to-End Training of Deep Visuomotor Policies
• Emergence of Locomotion Behaviours in Rich Environments
• Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization
• Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
• Generative Adversarial Imitation Learning
• One-Shot Visual Imitation Learning via Meta-Learning
• One-Shot Imitation Learning
• Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World
• Learning human behaviors from motion capture by adversarial imitation
• InfoGAIL: Interpretable Imitation Learning from Visual Demonstrations
• Collective Robot Reinforcement Learning with Distributed Asynchronous Guided Policy Search
• Transferring End-to-End Visuomotor Control from Simulation to Real World for a Multi-Stage Task
• Data-efficient Deep Reinforcement Learning for Dexterous Manipulation
• Learning and Transfer of Modulated Locomotor Controllers
• Learning a visuomotor controller for real world robotic grasping using simulated depth images
• Time-Contrastive Networks: Self-Supervised Learning from Video
• Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updates
• Learning Dexterous Manipulation Policies from Experience and Imitation
• Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translation
• Learning Dexterous Manipulation for a Soft Robotic Hand from Human Demonstration

Cited by in corpus (47)

• Solving Rubik's Cube with a Robot Hand
• Convolutional Neural Networks as a Model of the Visual System: Past, Present, and Future
• Learning Dexterous In-Hand Manipulation
• dm_control: Software and Tasks for Continuous Control
• PRIMAL2: Pathfinding via Reinforcement and Imitation Multi-Agent Learning -- Lifelong
• A Minimalist Approach to Offline Reinforcement Learning
• RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitation
• Reward learning from human preferences and demonstrations in Atari
• A Divergence Minimization Perspective on Imitation Learning Methods
• Visual Imitation Made Easy
• Toward the Fundamental Limits of Imitation Learning
• Dexterous Manipulation with Deep Reinforcement Learning: Efficient, General, and Low-Cost
• Driving Decision and Control for Autonomous Lane Change based on Deep Reinforcement Learning
• AC-Teach: A Bayesian Actor-Critic Method for Policy Learning with an Ensemble of Suboptimal Teachers
• Modelling Generalized Forces with Reinforcement Learning for Sim-to-Real Transfer
• Q-Learning for Continuous Actions with Cross-Entropy Guided Policies
• Motion Planner Augmented Reinforcement Learning for Robot Manipulation in Obstructed Environments
• Goal-Auxiliary Actor-Critic for 6D Robotic Grasping with Point Clouds
• Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning
• Residual Reinforcement Learning from Demonstrations
• From explanation to synthesis: Compositional program induction for learning from demonstration
• Automata Guided Reinforcement Learning With Demonstrations
• Human-in-the-Loop Imitation Learning using Remote Teleoperation
• Generalization Guarantees for Imitation Learning
• Understanding Multi-Modal Perception Using Behavioral Cloning for Peg-In-a-Hole Insertion Tasks
• Learning Dense Rewards for Contact-Rich Manipulation Tasks
• Adaptive Online Planning for Continual Lifelong Learning
• Sim2Real for Peg-Hole Insertion with Eye-in-Hand Camera
• Task-Oriented Hand Motion Retargeting for Dexterous Manipulation Imitation
• Hindsight Generative Adversarial Imitation Learning
• Adversarial Skill Chaining for Long-Horizon Robot Manipulation via Terminal State Regularization
• Robust Multi-Modal Policies for Industrial Assembly via Reinforcement Learning and Demonstrations: A Large-Scale Study
• Reinforced Imitation in Heterogeneous Action Space
• Imitation Learning with Sinkhorn Distances
• Reinforcement Learning-based Visual Navigation with Information-Theoretic Regularization
• Hybrid Reinforcement Learning with Expert State Sequences
• Cross-Domain Imitation Learning via Optimal Transport
• ShapeStacks: Learning Vision-Based Physical Intuition for Generalised Object Stacking
• Pre-training of Deep RL Agents for Improved Learning under Domain Randomization
• Pay attention! - Robustifying a Deep Visuomotor Policy through Task-Focused Attention
• Value-driven Hindsight Modelling
• Forgetful Experience Replay in Hierarchical Reinforcement Learning from Demonstrations
• "What, not how": Solving an under-actuated insertion task from scratch
• A dataset of 40K naturalistic 6-degree-of-freedom robotic grasp demonstrations
• Hybrid system identification using switching density networks
• When Autonomous Systems Meet Accuracy and Transferability through AI: A Survey
• Integration of Imitation Learning using GAIL and Reinforcement Learning using Task-achievement Rewards via Probabilistic Graphical Model