Virtual to Real Reinforcement Learning for Autonomous Driving

Abstract

Reinforcement learning is considered as a promising direction for driving policy learning. However, training autonomous driving vehicle with reinforcement learning in real environment involves non-affordable trial-and-error. It is more desirable to first train in a virtual environment and then transfer to the real environment. In this paper, we propose a novel realistic translation network to make model trained in virtual environment be workable in real world. The proposed network can convert non-realistic virtual image input into a realistic one with similar scene structure. Given realistic frames as input, driving policy trained by reinforcement learning can nicely adapt to real world driving. Experiments show that our proposed virtual to real (VR) reinforcement learning (RL) works pretty well. To our knowledge, this is the first successful case of driving policy trained by reinforcement learning that can adapt to real world driving data.

References in corpus (4)

• Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
• Safe, Multi-Agent, Reinforcement Learning for Autonomous Driving
• Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World
• End-to-End Deep Reinforcement Learning for Lane Keeping Assist

Cited by in corpus (48)

• An Introduction to Deep Reinforcement Learning
• A review of domain adaptation without target labels
• A Survey of End-to-End Driving: Architectures and Training Methods
• One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learning
• Machine Learning Testing: Survey, Landscapes and Horizons
• Learning to Walk in the Real World with Minimal Human Effort
• Empowering Things with Intelligence: A Survey of the Progress, Challenges, and Opportunities in Artificial Intelligence of Things
• Yes, we GAN: Applying Adversarial Techniques for Autonomous Driving
• Deep Reinforcement Learning for Autonomous Driving
• Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations
• OpenStreetMap-based Autonomous Navigation With LiDAR Naive-Valley-Path Obstacle Avoidance
• Predicting drag on rough surfaces by transfer learning of empirical correlations
• Autonomous Driving in Reality with Reinforcement Learning and Image Translation
• Improving Generalization of Transfer Learning Across Domains Using Spatio-Temporal Features in Autonomous Driving
• DeepRacing: Parameterized Trajectories for Autonomous Racing
• Fever Basketball: A Complex, Flexible, and Asynchronized Sports Game Environment for Multi-agent Reinforcement Learning
• Latent Space Reinforcement Learning for Steering Angle Prediction
• Safe Reinforcement Learning for Autonomous Vehicles through Parallel Constrained Policy Optimization
• Two-stage Deep Reinforcement Learning for Inverter-based Volt-VAR Control in Active Distribution Networks
• Hierarchical Policy for Non-prehensile Multi-object Rearrangement with Deep Reinforcement Learning and Monte Carlo Tree Search
• A Survey of Deep Reinforcement Learning Algorithms for Motion Planning and Control of Autonomous Vehicles
• Modular Vehicle Control for Transferring Semantic Information Between Weather Conditions Using GANs
• Domain Adaptation In Reinforcement Learning Via Latent Unified State Representation
• Reinforcement Learning based Control of Imitative Policies for Near-Accident Driving
• Driving Experience Transfer Method for End-to-End Control of Self-Driving Cars
• End-to-end Interpretable Neural Motion Planner
• PhysGAN: Generating Physical-World-Resilient Adversarial Examples for Autonomous Driving
• CIRL: Controllable Imitative Reinforcement Learning for Vision-based Self-driving
• Relationship Explainable Multi-objective Reinforcement Learning with Semantic Explainability Generation
• WAD: A Deep Reinforcement Learning Agent for Urban Autonomous Driving
• SENTINEL: Taming Uncertainty with Ensemble-based Distributional Reinforcement Learning
• Auto-Agent-Distiller: Towards Efficient Deep Reinforcement Learning Agents via Neural Architecture Search
• Learning to Navigate from Simulation via Spatial and Semantic Information Synthesis with Noise Model Embedding
• Relationship Explainable Multi-objective Optimization Via Vector Value Function Based Reinforcement Learning
• A coevolutionary approach to deep multi-agent reinforcement learning
• Transfer Learning and Organic Computing for Autonomous Vehicles
• Integrating Imitation Learning with Human Driving Data into Reinforcement Learning to Improve Training Efficiency for Autonomous Driving
• A Safe Hierarchical Planning Framework for Complex Driving Scenarios based on Reinforcement Learning
• Instance-Aware Predictive Navigation in Multi-Agent Environments
• Imitation Learning Approach for AI Driving Olympics Trained on Real-world and Simulation Data Simultaneously
• Enhanced Transfer Learning for Autonomous Driving with Systematic Accident Simulation
• Mutation Testing framework for Machine Learning
• A Virtual Testbed for Critical Incident Investigation with Autonomous Remote Aerial Vehicle Surveying, Artificial Intelligence, and Decision Support
• cGANs for Cartoon to Real-life Images
• Playing optical tweezers with deep reinforcement learning: in virtual, physical and augmented environments
• Partially fake it till you make it: mixing real and fake thermal images for improved object detection
• Curiosity Killed or Incapacitated the Cat and the Asymptotically Optimal Agent
• Towards a Sample Efficient Reinforcement Learning Pipeline for Vision Based Robotics