Learning Finite State Representations of Recurrent Policy Networks

Abstract

Recurrent neural networks (RNNs) are an effective representation of control policies for a wide range of reinforcement and imitation learning problems. RNN policies, however, are particularly difficult to explain, understand, and analyze due to their use of continuous-valued memory vectors and observation features. In this paper, we introduce a new technique, Quantized Bottleneck Insertion, to learn finite representations of these vectors and features. The result is a quantized representation of the RNN that can be analyzed to improve our understanding of memory use and general behavior. We present results of this approach on synthetic environments and six Atari games. The resulting finite representations are surprisingly small in some cases, using as few as 3 discrete memory states and 10 observations for a perfect Pong policy. We also show that these finite policy representations lead to improved interpretability.

Preprint. Under review at ICLR 2019

Cited by in corpus (9)

• Understanding the Role of Individual Units in a Deep Neural Network
• Interestingness Elements for Explainable Reinforcement Learning: Understanding Agents' Capabilities and Limitations
• Counterfactual State Explanations for Reinforcement Learning Agents via Generative Deep Learning
• Induction and Exploitation of Subgoal Automata for Reinforcement Learning
• The Emerging Landscape of Explainable AI Planning and Decision Making
• Learning Neural-Symbolic Descriptive Planning Models via Cube-Space Priors: The Voyage Home (to STRIPS)
• Generating Probabilistic Safety Guarantees for Neural Network Controllers
• Neural-Symbolic Descriptive Action Model from Images: The Search for STRIPS
• A2: Extracting Cyclic Switchings from DOB-nets for Rejecting Excessive Disturbances