Interestingness Elements for Explainable Reinforcement Learning: Understanding Agents' Capabilities and Limitations

Abstract

We propose an explainable reinforcement learning (XRL) framework that analyzes an agent's history of interaction with the environment to extract interestingness elements that help explain its behavior. The framework relies on data readily available from standard RL algorithms, augmented with data that can easily be collected by the agent while learning. We describe how to create visual summaries of an agent's behavior in the form of short video-clips highlighting key interaction moments, based on the proposed elements. We also report on a user study where we evaluated the ability of humans to correctly perceive the aptitude of agents with different characteristics, including their capabilities and limitations, given visual summaries automatically generated by our framework. The results show that the diversity of aspects captured by the different interestingness elements is crucial to help humans correctly understand an agent's strengths and limitations in performing a task, and determine when it might need adjustments to improve its performance.

arXiv commentTo appear in: Artificial Intelligence

References in corpus (5)

• Experimental Comparison of Open Source Visual-Inertial-Based State Estimation Algorithms in the Underwater Domain
• Learning Latent Dynamics for Planning from Pixels
• Visualizing and Understanding Atari Agents
• Graying the black box: Understanding DQNs
• Learning Finite State Representations of Recurrent Policy Networks

Cited by in corpus (14)

• Explainable Deep Reinforcement Learning: State of the Art and Challenges
• A Survey on AI Assurance
• Acquisition of Chess Knowledge in AlphaZero
• Counterfactual State Explanations for Reinforcement Learning Agents via Generative Deep Learning
• Explainable robotic systems: Understanding goal-driven actions in a reinforcement learning scenario
• Analysing Deep Reinforcement Learning Agents Trained with Domain Randomisation
• Learning Symbolic Rules for Interpretable Deep Reinforcement Learning
• Explainable Goal-Driven Agents and Robots -- A Comprehensive Review
• AutoPreview: A Framework for Autopilot Behavior Understanding
• Explainable Autonomous Robots: A Survey and Perspective
• Beneficial and Harmful Explanatory Machine Learning
• Domain-Level Explainability -- A Challenge for Creating Trust in Superhuman AI Strategies
• Explaining Conditions for Reinforcement Learning Behaviors from Real and Imagined Data
• The Atari Data Scraper