DART: Noise Injection for Robust Imitation Learning

Abstract

One approach to Imitation Learning is Behavior Cloning, in which a robot observes a supervisor and infers a control policy. A known problem with this "off-policy" approach is that the robot's errors compound when drifting away from the supervisor's demonstrations. On-policy, techniques alleviate this by iteratively collecting corrective actions for the current robot policy. However, these techniques can be tedious for human supervisors, add significant computation burden, and may visit dangerous states during training. We propose an off-policy approach that injects noise into the supervisor's policy while demonstrating. This forces the supervisor to demonstrate how to recover from errors. We propose a new algorithm, DART (Disturbances for Augmenting Robot Trajectories), that collects demonstrations with injected noise, and optimizes the noise level to approximate the error of the robot's trained policy during data collection. We compare DART with DAgger and Behavior Cloning in two domains: in simulation with an algorithmic supervisor on the MuJoCo tasks (Walker, Humanoid, Hopper, Half-Cheetah) and in physical experiments with human supervisors training a Toyota HSR robot to perform grasping in clutter. For high dimensional tasks like Humanoid, DART can be up to $3x$ faster in computation time and only decreases the supervisor's cumulative reward by $5\%$ during training, whereas DAgger executes policies that have $80\%$ less cumulative reward than the supervisor. On the grasping in clutter task, DART obtains on average a $62\%$ performance increase over Behavior Cloning.

Cited by in corpus (34)

• BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning
• Learning to Optimize Join Queries With Deep Reinforcement Learning
• Iterative Reinforcement Learning Based Design of Dynamic Locomotion Skills for Cassie
• Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences
• Learning from Suboptimal Demonstration via Self-Supervised Reward Regression
• Better-than-Demonstrator Imitation Learning via Automatically-Ranked Demonstrations
• Forward and inverse reinforcement learning sharing network weights and hyperparameters
• ThriftyDAgger: Budget-Aware Novelty and Risk Gating for Interactive Imitation Learning
• Learning Robust Bed Making using Deep Imitation Learning with DART
• On the Sample Complexity of Stability Constrained Imitation Learning
• Deep Bayesian Reward Learning from Preferences
• CompILE: Compositional Imitation Learning and Execution
• Learning Multi-Stage Tasks with One Demonstration via Self-Replay
• Learning Self-Correctable Policies and Value Functions from Demonstrations with Negative Sampling
• Policy Learning Using Weak Supervision
• Seeing All the Angles: Learning Multiview Manipulation Policies for Contact-Rich Tasks from Demonstrations
• Learning by Cheating
• On Machine Learning and Structure for Mobile Robots
• Grasping with Chopsticks: Combating Covariate Shift in Model-free Imitation Learning for Fine Manipulation
• Learning Task Decomposition with Ordered Memory Policy Network
• Imitation Learning with Sinkhorn Distances
• TanksWorld: A Multi-Agent Environment for AI Safety Research
• Provably Breaking the Quadratic Error Compounding Barrier in Imitation Learning, Optimally
• LazyDAgger: Reducing Context Switching in Interactive Imitation Learning
• Fighting Copycat Agents in Behavioral Cloning from Observation Histories
• Reparameterized Variational Divergence Minimization for Stable Imitation
• Brain-Inspired Deep Imitation Learning for Autonomous Driving Systems
• Iterative Imitation Policy Improvement for Interactive Autonomous Driving
• PICO: Primitive Imitation for COntrol
• Constraint Estimation and Derivative-Free Recovery for Robot Learning from Demonstrations
• No Need for Interactions: Robust Model-Based Imitation Learning using Neural ODE
• Hierarchical Policies for Cluttered-Scene Grasping with Latent Plans
• Imitation Learning via Simultaneous Optimization of Policies and Auxiliary Trajectories
• Meta Adaptation using Importance Weighted Demonstrations