Finding and Visualizing Weaknesses of Deep Reinforcement Learning Agents
arXiv:1904.01318
Abstract
As deep reinforcement learning driven by visual perception becomes more widely used there is a growing need to better understand and probe the learned agents. Understanding the decision making process and its relationship to visual inputs can be very valuable to identify problems in learned behavior. However, this topic has been relatively under-explored in the research community. In this work we present a method for synthesizing visual inputs of interest for a trained agent. Such inputs or states could be situations in which specific actions are necessary. Further, critical states in which a very high or a very low reward can be achieved are often interesting to understand the situational awareness of the system as they can correspond to risky states. To this end, we learn a generative model over the state space of the environment and use its latent space to optimize a target function for the state of interest. In our experiments we show that this method can generate insights for a variety of environments and reinforcement learning methods. We explore results in the standard Atari benchmark games as well as in an autonomous driving simulator. Based on the efficiency with which we have been able to identify behavioural weaknesses with this technique, we believe this general approach could serve as an important tool for AI safety applications.
References in corpus (18)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networks
- Striving for Simplicity: The All Convolutional Net
- Dueling Network Architectures for Deep Reinforcement Learning
- Visualizing and Understanding Recurrent Networks
- Deep Image Prior
- Visualizing Deep Convolutional Neural Networks Using Natural Pre-Images
- Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation
- Visualizing Deep Neural Network Decisions: Prediction Difference Analysis
- The (Un)reliability of saliency methods
- Multifaceted Feature Visualization: Uncovering the Different Types of Features Learned By Each Neuron in Deep Neural Networks
- Do Convnets Learn Correspondence?
- Graying the black box: Understanding DQNs
- A Taxonomy and Library for Visualizing Learned Features in Convolutional Neural Networks
- Local Explanation Methods for Deep Neural Networks Lack Sensitivity to Parameter Values
- Understanding Regularization to Visualize Convolutional Neural Networks