Deep Latent Competition: Learning to Race Using Visual Control Policies in Latent Space

Abstract

Learning competitive behaviors in multi-agent settings such as racing requires long-term reasoning about potential adversarial interactions. This paper presents Deep Latent Competition (DLC), a novel reinforcement learning algorithm that learns competitive visual control policies through self-play in imagination. The DLC agent imagines multi-agent interaction sequences in the compact latent space of a learned world model that combines a joint transition function with opponent viewpoint prediction. Imagined self-play reduces costly sample generation in the real world, while the latent representation enables planning to scale gracefully with observation dimensionality. We demonstrate the effectiveness of our algorithm in learning competitive behaviors on a novel multi-agent racing benchmark that requires planning from image observations. Code and videos available at https://sites.google.com/view/deep-latent-competition.

arXiv commentWilko, Tim, and Igor contributed equally to this work; published in Conference on Robot Learning 2020

References in corpus (13)

• End to End Learning for Self-Driving Cars
• DeepStack: Expert-Level Artificial Intelligence in No-Limit Poker
• StarCraft II: A New Challenge for Reinforcement Learning
• Value-Function Approximations for Partially Observable Markov Decision Processes
• Recurrent World Models Facilitate Policy Evolution
• Learning Latent Dynamics for Planning from Pixels
• Deep Reinforcement Learning from Self-Play in Imperfect-Information Games
• SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning
• ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst
• Emergent Coordination Through Competition
• Real-Time Control for Autonomous Racing Based on Viability Theory
• Autonomous Racing with AutoRally Vehicles and Differential Games
• Learning to Plan Optimistically: Uncertainty-Guided Deep Exploration via Latent Model Ensembles