Uncertainty-Aware Reinforcement Learning for Collision Avoidance

Abstract

Reinforcement learning can enable complex, adaptive behavior to be learned automatically for autonomous robotic platforms. However, practical deployment of reinforcement learning methods must contend with the fact that the training process itself can be unsafe for the robot. In this paper, we consider the specific case of a mobile robot learning to navigate an a priori unknown environment while avoiding collisions. In order to learn collision avoidance, the robot must experience collisions at training time. However, high-speed collisions, even at training time, could damage the robot. A successful learning method must therefore proceed cautiously, experiencing only low-speed collisions until it gains confidence. To this end, we present an uncertainty-aware model-based learning algorithm that estimates the probability of collision together with a statistical estimate of uncertainty. By formulating an uncertainty-dependent cost function, we show that the algorithm naturally chooses to proceed cautiously in unfamiliar environments, and increases the velocity of the robot in settings where it has high confidence. Our predictive model is based on bootstrapped neural networks using dropout, allowing it to process raw sensory inputs from high-bandwidth sensors such as cameras. Our experimental evaluation demonstrates that our method effectively minimizes dangerous collisions at training time in an obstacle avoidance task for a simulated and real-world quadrotor, and a real-world RC car. Videos of the experiments can be found at https://sites.google.com/site/probcoll.

References in corpus (4)

• Continuous control with deep reinforcement learning
• Playing Atari with Deep Reinforcement Learning
• Deep Exploration via Bootstrapped DQN
• Random feedback weights support learning in deep neural networks

Cited by in corpus (69)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control
• A Survey of Deep RL and IL for Autonomous Driving Policy Learning
• MagNet: a Two-Pronged Defense against Adversarial Examples
• Go-Explore: a New Approach for Hard-Exploration Problems
• Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog
• Self-Supervised Visual Planning with Temporal Skip Connections
• Model-Predictive Policy Learning with Uncertainty Regularization for Driving in Dense Traffic
• A Systematic Comparison of Bayesian Deep Learning Robustness in Diabetic Retinopathy Tasks
• CrowdMove: Autonomous Mapless Navigation in Crowded Scenarios
• Certified Adversarial Robustness for Deep Reinforcement Learning
• Variational Inference MPC for Bayesian Model-based Reinforcement Learning
• Combining Optimal Control and Learning for Visual Navigation in Novel Environments
• Towards Optimally Decentralized Multi-Robot Collision Avoidance via Deep Reinforcement Learning
• Optimizing Collision Avoidance in Dense Airspace using Deep Reinforcement Learning
• Bridging adaptive management and reinforcement learning for more robust decisions
• Uncertainty estimation of pedestrian future trajectory using Bayesian approximation
• Unifying Cardiovascular Modelling with Deep Reinforcement Learning for Uncertainty Aware Control of Sepsis Treatment
• Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learning
• The AdobeIndoorNav Dataset: Towards Deep Reinforcement Learning based Real-world Indoor Robot Visual Navigation
• Mutual Alignment Transfer Learning
• Dropout Q-Functions for Doubly Efficient Reinforcement Learning
• The Emotionally Intelligent Robot: Improving Social Navigation in Crowded Environments
• Confidence Scoring Using Whitebox Meta-models with Linear Classifier Probes
• Navigating Assistance System for Quadcopter with Deep Reinforcement Learning
• AirSim Drone Racing Lab
• Ctrl-Z: Recovering from Instability in Reinforcement Learning
• Observation Space Matters: Benchmark and Optimization Algorithm
• Reset-Free Lifelong Learning with Skill-Space Planning
• Explaining Predictive Uncertainty by Exposing Second-Order Effects
• A Survey of Behavior Learning Applications in Robotics -- State of the Art and Perspectives
• Discovering Blind Spots in Reinforcement Learning
• One Solution is Not All You Need: Few-Shot Extrapolation via Structured MaxEnt RL
• PlaNet of the Bayesians: Reconsidering and Improving Deep Planning Network by Incorporating Bayesian Inference
• Generalizing from a few environments in safety-critical reinforcement learning
• Design Space of Behaviour Planning for Autonomous Driving
• Bayesian Inference with Certifiable Adversarial Robustness
• Safety Guarantees for Planning Based on Iterative Gaussian Processes
• Generating Robust Supervision for Learning-Based Visual Navigation Using Hamilton-Jacobi Reachability
• Learning Vision-based Reactive Policies for Obstacle Avoidance
• Regularizing Trajectory Optimization with Denoising Autoencoders
• AI based Algorithms of Path Planning, Navigation and Control for Mobile Ground Robots and UAVs
• A Multi-Agent Approach for Adaptive Finger Cooperation in Learning-based In-Hand Manipulation
• Offline reinforcement learning with uncertainty for treatment strategies in sepsis
• Danger-aware Adaptive Composition of DRL Agents for Self-navigation
• Evolutionary Computation and AI Safety: Research Problems Impeding Routine and Safe Real-world Application of Evolution
• Visual-based Autonomous Driving Deployment from a Stochastic and Uncertainty-aware Perspective
• CRAVES: Controlling Robotic Arm with a Vision-based Economic System
• Limits of Probabilistic Safety Guarantees when Considering Human Uncertainty
• GAN-based Priors for Quantifying Uncertainty
• Risk-Aware Planning by Confidence Estimation using Deep Learning-Based Perception
• SeRO: Self-Supervised Reinforcement Learning for Recovery from Out-of-Distribution Situations
• LS3: Latent Space Safe Sets for Long-Horizon Visuomotor Control of Sparse Reward Iterative Tasks
• Improving Model-Based Control and Active Exploration with Reconstruction Uncertainty Optimization
• Fully Bayesian Recurrent Neural Networks for Safe Reinforcement Learning
• Safe Policy Search with Gaussian Process Models
• Towards Physically Safe Reinforcement Learning under Supervision
• Safety Enhancement for Deep Reinforcement Learning in Autonomous Separation Assurance
• Learning to Be Cautious
• Deep Reinforcement Learning with Weighted Q-Learning
• DESTA: A Framework for Safe Reinforcement Learning with Markov Games of Intervention
• Learning Resilient Behaviors for Navigation Under Uncertainty
• Neural Flocking: MPC-based Supervised Learning of Flocking Controllers
• Lyapunov-based uncertainty-aware safe reinforcement learning
• Learning to See before Learning to Act: Visual Pre-training for Manipulation
• DeepMNavigate: Deep Reinforced Multi-Robot Navigation Unifying Local & Global Collision Avoidance
• Depth by Poking: Learning to Estimate Depth from Self-Supervised Grasping
• Toward Verifiable Real-Time Obstacle Motion Prediction for Dynamic Collision Avoidance
• On Offline Evaluation of Vision-based Driving Models