Graying the black box: Understanding DQNs

Abstract

In recent years there is a growing interest in using deep representations for reinforcement learning. In this paper, we present a methodology and tools to analyze Deep Q-networks (DQNs) in a non-blind matter. Moreover, we propose a new model, the Semi Aggregated Markov Decision Process (SAMDP), and an algorithm that learns it automatically. The SAMDP model allows us to identify spatio-temporal abstractions directly from features and may be used as a sub-goal detector in future work. Using our tools we reveal that the features learned by DQNs aggregate the state space in a hierarchical fashion, explaining its success. Moreover, we are able to understand and describe the policies learned by DQNs for three different Atari2600 games and suggest ways to interpret, debug and optimize deep neural networks in reinforcement learning.

References in corpus (8)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• How transferable are features in deep neural networks?
• Dueling Network Architectures for Deep Reinforcement Learning
• Mayavi: a package for 3D visualization of scientific data
• Massively Parallel Methods for Deep Reinforcement Learning
• Hierarchical Solution of Markov Decision Processes using Macro-actions
• Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning
• Flexible Decomposition Algorithms for Weakly Coupled Markov Decision Problems

Cited by in corpus (26)

• Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications
• A Survey and Critique of Multiagent Deep Reinforcement Learning
• Learn What Not to Learn: Action Elimination with Deep Reinforcement Learning
• Interestingness Elements for Explainable Reinforcement Learning: Understanding Agents' Capabilities and Limitations
• Explainable Artificial Intelligence: a Systematic Review
• Local and Global Explanations of Agent Behavior: Integrating Strategy Summaries with Saliency Maps
• DRLViz: Understanding Decisions and Memory in Deep Reinforcement Learning
• Explain Your Move: Understanding Agent Actions Using Specific and Relevant Feature Attribution
• Operationally meaningful representations of physical systems in neural networks
• Meta-trained agents implement Bayes-optimal agents
• The Emerging Landscape of Explainable AI Planning and Decision Making
• DeepView: Visualizing Classification Boundaries of Deep Neural Networks as Scatter Plots Using Discriminative Dimensionality Reduction
• Counterfactual States for Atari Agents via Generative Deep Learning
• Learn to Interpret Atari Agents
• An initial attempt of combining visual selective attention with deep reinforcement learning
• Don't Explain without Verifying Veracity: An Evaluation of Explainable AI with Video Activity Recognition
• Faster and Safer Training by Embedding High-Level Knowledge into Deep Reinforcement Learning
• Finding and Visualizing Weaknesses of Deep Reinforcement Learning Agents
• Learning Symbolic Rules for Interpretable Deep Reinforcement Learning
• Beneficial and Harmful Explanatory Machine Learning
• Are Gradient-based Saliency Maps Useful in Deep Reinforcement Learning?
• Towards Robust Explanations for Deep Neural Networks
• RoCUS: Robot Controller Understanding via Sampling
• How Do You Act? An Empirical Study to Understand Behavior of Deep Reinforcement Learning Agents
• Exploiting Language Instructions for Interpretable and Compositional Reinforcement Learning
• RL agents Implicitly Learning Human Preferences