DRLViz: Understanding Decisions and Memory in Deep Reinforcement Learning

Abstract

We present DRLViz, a visual analytics interface to interpret the internal memory of an agent (e.g. a robot) trained using deep reinforcement learning. This memory is composed of large temporal vectors updated when the agent moves in an environment and is not trivial to understand due to the number of dimensions, dependencies to past vectors, spatial/temporal correlations, and co-correlation between dimensions. It is often referred to as a black box as only inputs (images) and outputs (actions) are intelligible for humans. Using DRLViz, experts are assisted to interpret decisions using memory reduction interactions, and to investigate the role of parts of the memory when errors have been made (e.g. wrong direction). We report on DRLViz applied in the context of video games simulators (ViZDoom) for a navigation scenario with item gathering tasks. We also report on experts evaluation using DRLViz, and applicability of DRLViz to other scenarios and navigation problems beyond simulation games, as well as its contribution to black box models interpretability and explainability in the field of visual analytics.

References in corpus (12)

• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• LSTM: A Search Space Odyssey
• Playing Atari with Deep Reinforcement Learning
• A Brief Survey of Deep Reinforcement Learning
• End-to-End Training of Deep Visuomotor Policies
• The Arcade Learning Environment: An Evaluation Platform for General Agents
• Visualizing and Understanding Recurrent Networks
• RetainVis: Visual Analytics with Interpretable and Interactive Recurrent Neural Networks on Electronic Medical Records
• Playing FPS Games with Deep Reinforcement Learning
• Graying the black box: Understanding DQNs
• Understanding Hidden Memories of Recurrent Neural Networks
• MUREL: Multimodal Relational Reasoning for Visual Question Answering

Cited by in corpus (7)

• Analysing Deep Reinforcement Learning Agents Trained with Domain Randomisation
• A Survey of Visual Analytics Techniques for Machine Learning
• AutoDOViz: Human-Centered Automation for Decision Optimization
• MARLens: Understanding Multi-agent Reinforcement Learning for Traffic Signal Control via Visual Analytics
• Reinforced Labels: Multi-Agent Deep Reinforcement Learning for Point-Feature Label Placement
• SIM2REALVIZ: Visualizing the Sim2Real Gap in Robot Ego-Pose Estimation
• Why? Why not? When? Visual Explanations of Agent Behavior in Reinforcement Learning