FormulaZero: Distributionally Robust Online Adaptation via Offline Population Synthesis
arXiv:2003.03900
Abstract
Balancing performance and safety is crucial to deploying autonomous vehicles in multi-agent environments. In particular, autonomous racing is a domain that penalizes safe but conservative policies, highlighting the need for robust, adaptive strategies. Current approaches either make simplifying assumptions about other agents or lack robust mechanisms for online adaptation. This work makes algorithmic contributions to both challenges. First, to generate a realistic, diverse set of opponents, we develop a novel method for self-play based on replica-exchange Markov chain Monte Carlo. Second, we propose a distributionally robust bandit optimization procedure that adaptively adjusts risk aversion relative to uncertainty in beliefs about opponents' behaviors. We rigorously quantify the tradeoffs in performance and robustness when approximating these computations in real-time motion-planning, and we demonstrate our methods experimentally on autonomous vehicles that achieve scaled speeds comparable to Formula One racecars.
arXiv commentICML 2020: https://icml.cc/virtual/2020/poster/6277
References in corpus (14)
- Adam: A Method for Stochastic Optimization
- Simulated Tempering: A New Monte Carlo Scheme
- Dota 2 with Large Scale Deep Reinforcement Learning
- Human-level performance in first-person multiplayer games with population-based deep reinforcement learning
- Determinantal point processes for machine learning
- Illuminating search spaces by mapping elites
- Robust Adversarial Reinforcement Learning
- Certifying Some Distributional Robustness with Principled Adversarial Training
- Emergent Complexity via Multi-Agent Competition
- Adversarial Policies: Attacking Deep Reinforcement Learning
- Efficient Black-box Assessment of Autonomous Vehicle Safety
- Regret Bounds for Robust Adaptive Control of the Linear Quadratic Regulator
- Autonomous Racing with AutoRally Vehicles and Differential Games
- Online Vehicle Trajectory Prediction using Policy Anticipation Network and Optimization-based Context Reasoning