Super-Human Performance in Gran Turismo Sport Using Deep Reinforcement Learning

Abstract

Autonomous car racing is a major challenge in robotics. It raises fundamental problems for classical approaches such as planning minimum-time trajectories under uncertain dynamics and controlling the car at the limits of its handling. Besides, the requirement of minimizing the lap time, which is a sparse objective, and the difficulty of collecting training data from human experts have also hindered researchers from directly applying learning-based approaches to solve the problem. In the present work, we propose a learning-based system for autonomous car racing by leveraging a high-fidelity physical car simulation, a course-progress proxy reward, and deep reinforcement learning. We deploy our system in Gran Turismo Sport, a world-leading car simulator known for its realistic physics simulation of different race cars and tracks, which is even used to recruit human race car drivers. Our trained policy achieves autonomous racing performance that goes beyond what had been achieved so far by the built-in AI, and, at the same time, outperforms the fastest driver in a dataset of over 50,000 human players.

arXiv commentAccepted for Publication at the IEEE Robotics and Automation Letters (RA-L) 2021, and International Conference on Robots and Automation (ICRA) 2021

References in corpus (6)

• A Survey of Deep Learning Techniques for Autonomous Driving
• Learning agile and dynamic motor skills for legged robots
• Learning Quadrupedal Locomotion over Challenging Terrain
• Optimization-Based Autonomous Racing of 1:43 Scale RC Cars
• Control of a Quadrotor with Reinforcement Learning
• High-speed Autonomous Drifting with Deep Reinforcement Learning

Cited by in corpus (10)

• Autonomous Vehicles on the Edge: A Survey on Autonomous Vehicle Racing
• Learning Minimum-Time Flight in Cluttered Environments
• Model- and Acceleration-based Pursuit Controller for High-Performance Autonomous Racing
• Safety-aware Causal Representation for Trustworthy Offline Reinforcement Learning in Autonomous Driving
• TC-Driver: Trajectory Conditioned Driving for Robust Autonomous Racing -- A Reinforcement Learning Approach
• Winning the 3rd Japan Automotive AI Challenge -- Autonomous Racing with the Autoware.Auto Open Source Software Stack
• Learning Autonomous Race Driving with Action Mapping Reinforcement Learning
• Mixed-Reality Digital Twins: Leveraging the Physical and Virtual Worlds for Hybrid Sim2Real Transition of Multi-Agent Reinforcement Learning Policies
• Learning Reward Machines in Cooperative Multi-Agent Tasks
• A Champion-level Vision-based Reinforcement Learning Agent for Competitive Racing in Gran Turismo 7