Deep Imitative Models for Flexible Inference, Planning, and Control

Abstract

Imitation Learning (IL) is an appealing approach to learn desirable autonomous behavior. However, directing IL to achieve arbitrary goals is difficult. In contrast, planning-based algorithms use dynamics models and reward functions to achieve goals. Yet, reward functions that evoke desirable behavior are often difficult to specify. In this paper, we propose Imitative Models to combine the benefits of IL and goal-directed planning. Imitative Models are probabilistic predictive models of desirable behavior able to plan interpretable expert-like trajectories to achieve specified goals. We derive families of flexible goal objectives, including constrained goal regions, unconstrained goal sets, and energy-based goals. We show that our method can use these objectives to successfully direct behavior. Our method substantially outperforms six IL approaches and a planning-based approach in a dynamic simulated autonomous driving task, and is efficiently learned from expert demonstrations without online data collection. We also show our approach is robust to poorly specified goals, such as goals on the wrong side of the road.

References in corpus (6)

• Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
• Differentiable MPC for End-to-end Planning and Control
• Reinforcement and Imitation Learning via Interactive No-Regret Learning
• Rethinking Self-driving: Multi-task Knowledge for Better Generalization and Accident Explanation Ability
• DropoutDAgger: A Bayesian Approach to Safe Imitation Learning
• Naturalistic Driver Intention and Path Prediction using Recurrent Neural Networks

Cited by in corpus (21)

• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• INTERACTION Dataset: An INTERnational, Adversarial and Cooperative moTION Dataset in Interactive Driving Scenarios with Semantic Maps
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning
• Multimodal End-to-End Autonomous Driving
• Social Attention for Autonomous Decision-Making in Dense Traffic
• Efficient Black-box Assessment of Autonomous Vehicle Safety
• Learn to Navigate Maplessly with Varied LiDAR Configurations: A Support Point-Based Approach
• PLOP: Probabilistic poLynomial Objects trajectory Planning for autonomous driving
• Deep Imitation Learning for Bimanual Robotic Manipulation
• ObserveNet Control: A Vision-Dynamics Learning Approach to Predictive Control in Autonomous Vehicles
• Multi-Agent Reinforcement Learning with Multi-Step Generative Models
• Multimodal Trajectory Prediction via Topological Invariance for Navigation at Uncontrolled Intersections
• Affordance-based Reinforcement Learning for Urban Driving
• A Survey of Deep Reinforcement Learning Algorithms for Motion Planning and Control of Autonomous Vehicles
• Data-efficient visuomotor policy training using reinforcement learning and generative models
• Safe Trajectory Planning Using Reinforcement Learning for Self Driving
• Zero-shot Imitation Learning from Demonstrations for Legged Robot Visual Navigation
• SAFARI: Safe and Active Robot Imitation Learning with Imagination
• Reinforced Imitation Learning by Free Energy Principle
• Learning a Decision Module by Imitating Driver's Control Behaviors
• 3rd Place Solution for NeurIPS 2021 Shifts Challenge: Vehicle Motion Prediction