Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards

Abstract

We propose a general and model-free approach for Reinforcement Learning (RL) on real robotics with sparse rewards. We build upon the Deep Deterministic Policy Gradient (DDPG) algorithm to use demonstrations. Both demonstrations and actual interactions are used to fill a replay buffer and the sampling ratio between demonstrations and transitions is automatically tuned via a prioritized replay mechanism. Typically, carefully engineered shaping rewards are required to enable the agents to efficiently explore on high dimensional control problems such as robotics. They are also required for model-based acceleration methods relying on local solvers such as iLQG (e.g. Guided Policy Search and Normalized Advantage Function). The demonstrations replace the need for carefully engineered rewards, and reduce the exploration problem encountered by classical RL approaches in these domains. Demonstrations are collected by a robot kinesthetically force-controlled by a human demonstrator. Results on four simulated insertion tasks show that DDPG from demonstrations out-performs DDPG, and does not require engineered rewards. Finally, we demonstrate the method on a real robotics task consisting of inserting a clip (flexible object) into a rigid object.

References in corpus (3)

• Continuous control with deep reinforcement learning
• Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization
• Generative Adversarial Imitation Learning

Cited by in corpus (145)

• How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned
• Conservative Q-Learning for Offline Reinforcement Learning
• Deep Reinforcement Learning: An Overview
• Exploration in Deep Reinforcement Learning: A Survey
• PRIMAL: Pathfinding via Reinforcement and Imitation Multi-Agent Learning
• Off-Policy Deep Reinforcement Learning without Exploration
• Learning by Playing - Solving Sparse Reward Tasks from Scratch
• Reinforcement Learning for IoT Security: A Comprehensive Survey
• Deep Reinforcement Learning for the Control of Robotic Manipulation: A Focussed Mini-Review
• A Minimalist Approach to Offline Reinforcement Learning
• Transfer Learning in Deep Reinforcement Learning: A Survey
• Prioritized Experience-based Reinforcement Learning with Human Guidance for Autonomous Driving
• Reinforcement and Imitation Learning for Diverse Visuomotor Skills
• Sim-to-Real Reinforcement Learning for Deformable Object Manipulation
• Compare Contact Model-based Control and Contact Model-free Learning: A Survey of Robotic Peg-in-hole Assembly Strategies
• Observe and Look Further: Achieving Consistent Performance on Atari
• Automatic spin-chain learning to explore the quantum speed limit
• A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation
• AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
• Data-driven control of room temperature and bidirectional EV charging using deep reinforcement learning: simulations and experiments
• Residual Learning from Demonstration: Adapting DMPs for Contact-rich Manipulation
• Making Efficient Use of Demonstrations to Solve Hard Exploration Problems
• Modified DDPG car-following model with a real-world human driving experience with CARLA simulator
• Scaling data-driven robotics with reward sketching and batch reinforcement learning
• CURIOUS: Intrinsically Motivated Modular Multi-Goal Reinforcement Learning
• RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitation
• Reward learning from human preferences and demonstrations in Atari
• Demonstration-Guided Deep Reinforcement Learning of Control Policies for Dexterous Human-Robot Interaction
• Reinforcement Learning for Robotic Manipulation using Simulated Locomotion Demonstrations
• From Screens to Scenes: A Survey of Embodied AI in Healthcare
• Overcoming Exploration: Deep Reinforcement Learning for Continuous Control in Cluttered Environments from Temporal Logic Specifications
• Temporal Logic Guided Safe Reinforcement Learning Using Control Barrier Functions
• Goal-conditioned Imitation Learning
• Human-in-the-Loop Deep Reinforcement Learning with Application to Autonomous Driving
• Parrot: Data-Driven Behavioral Priors for Reinforcement Learning
• Learning to Reach Goals via Iterated Supervised Learning
• The Ingredients of Real-World Robotic Reinforcement Learning
• Active Deep Q-learning with Demonstration
• Deep Reinforcement Learning based Local Planner for UAV Obstacle Avoidance using Demonstration Data
• Learning Self-Imitating Diverse Policies
• Knowledge Transfer for Cross-Domain Reinforcement Learning: A Systematic Review
• Asymmetric self-play for automatic goal discovery in robotic manipulation
• CACTO: Continuous Actor-Critic with Trajectory Optimization -- Towards global optimality
• Sample-Efficient Imitation Learning via Generative Adversarial Nets
• Learning latent state representation for speeding up exploration
• VisuoSpatial Foresight for Multi-Step, Multi-Task Fabric Manipulation
• Constrained-Space Optimization and Reinforcement Learning for Complex Tasks
• Dexterous Manipulation with Deep Reinforcement Learning: Efficient, General, and Low-Cost
• AC-Teach: A Bayesian Actor-Critic Method for Policy Learning with an Ensemble of Suboptimal Teachers
• Safety-Guided Deep Reinforcement Learning via Online Gaussian Process Estimation
• Learning Personalized Human-Aware Robot Navigation Using Virtual Reality Demonstrations from a User Study
• Reinforcement Learning on Web Interfaces Using Workflow-Guided Exploration
• Stellar Spectral Interpolation using Machine Learning
• Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills
• Offline Learning from Demonstrations and Unlabeled Experience
• Goal-Auxiliary Actor-Critic for 6D Robotic Grasping with Point Clouds
• Learning Visual Robotic Control Efficiently with Contrastive Pre-training and Data Augmentation
• Learning Dexterous Manipulation from Suboptimal Experts
• Revisiting Design Choices in Proximal Policy Optimization
• Learning Object Manipulation Skills via Approximate State Estimation from Real Videos
• On Multi-Agent Learning in Team Sports Games
• Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble
• Efficient Deep Reinforcement Learning with Imitative Expert Priors for Autonomous Driving
• HRL4IN: Hierarchical Reinforcement Learning for Interactive Navigation with Mobile Manipulators
• Curiosity-Driven Multi-Criteria Hindsight Experience Replay
• Dynamic Pricing on E-commerce Platform with Deep Reinforcement Learning: A Field Experiment
• Residual Reinforcement Learning from Demonstrations
• Scaling Robot Supervision to Hundreds of Hours with RoboTurk: Robotic Manipulation Dataset through Human Reasoning and Dexterity
• Learning Sparse Rewarded Tasks from Sub-Optimal Demonstrations
• Cloud-Edge Training Architecture for Sim-to-Real Deep Reinforcement Learning
• Automata Guided Reinforcement Learning With Demonstrations
• An Equivalence between Loss Functions and Non-Uniform Sampling in Experience Replay
• Dynamic Experience Replay
• Self-Supervised Sim-to-Real Adaptation for Visual Robotic Manipulation
• Reinforcement Learning from Imperfect Demonstrations under Soft Expert Guidance
• Demonstration-Guided Reinforcement Learning with Learned Skills
• Causal-aware Safe Policy Improvement for Task-oriented dialogue
• Integrating Behavior Cloning and Reinforcement Learning for Improved Performance in Dense and Sparse Reward Environments
• Learning from Guided Play: Improving Exploration for Adversarial Imitation Learning with Simple Auxiliary Tasks
• Learning Multi-Stage Tasks with One Demonstration via Self-Replay
• Capability Iteration Network for Robot Path Planning
• A Grasp Pose is All You Need: Learning Multi-fingered Grasping with Deep Reinforcement Learning from Vision and Touch
• ZPD Teaching Strategies for Deep Reinforcement Learning from Demonstrations
• Safety Augmented Value Estimation from Demonstrations (SAVED): Safe Deep Model-Based RL for Sparse Cost Robotic Tasks
• Learning Self-Correctable Policies and Value Functions from Demonstrations with Negative Sampling
• VisuoSpatial Foresight for Physical Sequential Fabric Manipulation
• Using Offline Data to Speed Up Reinforcement Learning in Procedurally Generated Environments
• Reward Engineering for Object Pick and Place Training
• Flow Rate Control in Smart District Heating Systems Using Deep Reinforcement Learning
• Blockchain-assisted Demonstration Cloning for Multi-Agent Deep Reinforcement Learning
• Exploration-efficient Deep Reinforcement Learning with Demonstration Guidance for Robot Control
• Expert-augmented actor-critic for ViZDoom and Montezumas Revenge
• Shaping Rewards for Reinforcement Learning with Imperfect Demonstrations using Generative Models
• Vision-Based Autonomous Car Racing Using Deep Imitative Reinforcement Learning
• PsiPhi-Learning: Reinforcement Learning with Demonstrations using Successor Features and Inverse Temporal Difference Learning
• How Much Do Unstated Problem Constraints Limit Deep Robotic Reinforcement Learning?
• Balance Between Efficient and Effective Learning: Dense2Sparse Reward Shaping for Robot Manipulation with Environment Uncertainty
• Reinforcement Learning with Videos: Combining Offline Observations with Interaction
• Deep Reinforcement Learning with Feedback-based Exploration
• RL-Controller: a reinforcement learning framework for active structural control
• Learning from Ambiguous Demonstrations with Self-Explanation Guided Reinforcement Learning
• Robust Multi-Modal Policies for Industrial Assembly via Reinforcement Learning and Demonstrations: A Large-Scale Study
• AW-Opt: Learning Robotic Skills with Imitation and Reinforcement at Scale
• Accelerating Safe Reinforcement Learning with Constraint-mismatched Policies
• Reinforced Imitation in Heterogeneous Action Space
• Trajectory Optimization for Unknown Constrained Systems using Reinforcement Learning
• A Learning Framework for High Precision Industrial Assembly
• Human-in-the-Loop Methods for Data-Driven and Reinforcement Learning Systems
• Reinforced Imitation Learning by Free Energy Principle
• Pretrain Soft Q-Learning with Imperfect Demonstrations
• Bridging the Imitation Gap by Adaptive Insubordination
• Auto-Agent-Distiller: Towards Efficient Deep Reinforcement Learning Agents via Neural Architecture Search
• DDPG++: Striving for Simplicity in Continuous-control Off-Policy Reinforcement Learning
• CIRL: Controllable Imitative Reinforcement Learning for Vision-based Self-driving
• Adaptive Procedural Task Generation for Hard-Exploration Problems
• Density-based Curriculum for Multi-goal Reinforcement Learning with Sparse Rewards
• Follow the Object: Curriculum Learning for Manipulation Tasks with Imagined Goals
• Improving Learning from Demonstrations by Learning from Experience
• Self-Imitation Learning by Planning
• Forgetful Experience Replay in Hierarchical Reinforcement Learning from Demonstrations
• Continuous Control with Action Quantization from Demonstrations
• Distilling Motion Planner Augmented Policies into Visual Control Policies for Robot Manipulation
• Episodic Self-Imitation Learning with Hindsight
• Bootstrapping Motor Skill Learning with Motion Planning
• Accelerated Sim-to-Real Deep Reinforcement Learning: Learning Collision Avoidance from Human Player
• Wish you were here: Hindsight Goal Selection for long-horizon dexterous manipulation
• Tolerance-Guided Policy Learning for Adaptable and Transferrable Delicate Industrial Insertion
• Goal-constrained Sparse Reinforcement Learning for End-to-End Driving
• What Robot do I Need? Fast Co-Adaptation of Morphology and Control using Graph Neural Networks
• SEIHAI: A Sample-efficient Hierarchical AI for the MineRL Competition
• DCUR: Data Curriculum for Teaching via Samples with Reinforcement Learning
• Homotopy Based Reinforcement Learning with Maximum Entropy for Autonomous Air Combat
• Parallelized Reverse Curriculum Generation
• Multi-Preference Actor Critic
• Hierarchical Policies for Cluttered-Scene Grasping with Latent Plans
• Targeted Data Acquisition for Evolving Negotiation Agents
• Automatic Curricula via Expert Demonstrations
• Learning with Training Wheels: Speeding up Training with a Simple Controller for Deep Reinforcement Learning
• Evaluating task-agnostic exploration for fixed-batch learning of arbitrary future tasks
• Cooperative Multi-Agent Policy Gradients with Sub-optimal Demonstration
• Efficiently Training On-Policy Actor-Critic Networks in Robotic Deep Reinforcement Learning with Demonstration-like Sampled Exploration
• Learning Memory-Dependent Continuous Control from Demonstrations
• State Representation Learning from Demonstration
• Explore and Control with Adversarial Surprise
• Solving the Real Robot Challenge using Deep Reinforcement Learning