Causal Confusion in Imitation Learning

Abstract

Behavioral cloning reduces policy learning to supervised learning by training a discriminative model to predict expert actions given observations. Such discriminative models are non-causal: the training procedure is unaware of the causal structure of the interaction between the expert and the environment. We point out that ignoring causality is particularly damaging because of the distributional shift in imitation learning. In particular, it leads to a counter-intuitive "causal misidentification" phenomenon: access to more information can yield worse performance. We investigate how this problem arises, and propose a solution to combat it through targeted interventions---either environment interaction or expert queries---to determine the correct causal model. We show that causal misidentification occurs in several benchmark control domains as well as realistic driving settings, and validate our solution against DAgger and other baselines and ablations.

arXiv commentPublished at NeurIPS 2019 9 pages, plus references and appendices

References in corpus (16)

• Playing Atari with Deep Reinforcement Learning
• Categorical Reparameterization with Gumbel-Softmax
• End to End Learning for Self-Driving Cars
• Trust Region Policy Optimization
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Variational Dropout and the Local Reparameterization Trick
• Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
• Causal Effect Inference with Deep Latent-Variable Models
• Understanding disentangling in $\beta$-VAE
• A fast PC algorithm for high dimensional causal discovery with multi-core PCs
• Isolating Sources of Disentanglement in Variational Autoencoders
• Estimating Mutual Information for Discrete-Continuous Mixtures
• ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst
• Causal Inference via Kernel Deviance Measures
• Identifying Best Interventions through Online Importance Sampling
• The Blessings of Multiple Causes

Cited by in corpus (28)

• A Survey of End-to-End Driving: Architectures and Training Methods
• Interventional Few-Shot Learning
• A causal view of compositional zero-shot recognition
• FusionPlanner: A Multi-task Motion Planner for Mining Trucks via Multi-sensor Fusion
• Modified DDPG car-following model with a real-world human driving experience with CARLA simulator
• Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences
• Invariant Risk Minimization Games
• Rethinking Closed-loop Training for Autonomous Driving
• Weakly Supervised Reinforcement Learning for Autonomous Highway Driving via Virtual Safety Cages
• Multimodal and Force-Matched Imitation Learning with a See-Through Visuotactile Sensor
• CARLA Real Traffic Scenarios -- novel training ground and benchmark for autonomous driving
• The NVIDIA PilotNet Experiments
• Multi-task Batch Reinforcement Learning with Metric Learning
• Who Make Drivers Stop? Towards Driver-centric Risk Assessment: Risk Object Identification via Causal Inference
• Grounded Relational Inference: Domain Knowledge Driven Explainable Autonomous Driving
• Action-Based Representation Learning for Autonomous Driving
• Resolving Spurious Correlations in Causal Models of Environments via Interventions
• RITA: Boost Driving Simulators with Realistic Interactive Traffic Flow
• Learning from Ambiguous Demonstrations with Self-Explanation Guided Reinforcement Learning
• Simulation-Based Counterfactual Causal Discovery on Real World Driver Behaviour
• Understanding Learned Reward Functions
• Preventing Imitation Learning with Adversarial Policy Ensembles
• Interpretable Imitation Learning with Dynamic Causal Relations
• Typing assumptions improve identification in causal discovery
• Deep PQR: Solving Inverse Reinforcement Learning using Anchor Actions
• UMBRELLA: Uncertainty-Aware Model-Based Offline Reinforcement Learning Leveraging Planning
• Less is More: Generating Grounded Navigation Instructions from Landmarks
• On complementing end-to-end human behavior predictors with planning