Programmatically Interpretable Reinforcement Learning

Abstract

We present a reinforcement learning framework, called Programmatically Interpretable Reinforcement Learning (PIRL), that is designed to generate interpretable and verifiable agent policies. Unlike the popular Deep Reinforcement Learning (DRL) paradigm, which represents policies by neural networks, PIRL represents policies using a high-level, domain-specific programming language. Such programmatic policies have the benefits of being more easily interpreted than neural networks, and being amenable to verification by symbolic methods. We propose a new method, called Neurally Directed Program Search (NDPS), for solving the challenging nonsmooth optimization problem of finding a programmatic policy with maximal reward. NDPS works by first learning a neural policy network using DRL, and then performing a local search over programmatic policies that seeks to minimize a distance from this neural "oracle". We evaluate NDPS on the task of learning to drive a simulated car in the TORCS car-racing environment. We demonstrate that NDPS is able to discover human-readable policies that pass some significant performance bars. We also show that PIRL policies can have smoother trajectories, and can be more easily transferred to environments not encountered during training, than corresponding policies discovered by DRL.

Published at The 35th International Conference on Machine Learning (ICML 2018)

References in corpus (9)

• Continuous control with deep reinforcement learning
• Practical Bayesian Optimization of Machine Learning Algorithms
• Methods for Interpreting and Understanding Deep Neural Networks
• Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
• DeepCoder: Learning to Write Programs
• Towards Deep Symbolic Reinforcement Learning
• Neural Sketch Learning for Conditional Program Generation
• Leveraging Grammar and Reinforcement Learning for Neural Program Synthesis
• Neural Program Synthesis with Priority Queue Training

Cited by in corpus (24)

• Challenges of Real-World Reinforcement Learning
• Verifiable Reinforcement Learning via Policy Extraction
• Interpreting Deep Learning-Based Networking Systems
• Counterfactual State Explanations for Reinforcement Learning Agents via Generative Deep Learning
• Interpretable Multi-Objective Reinforcement Learning through Policy Orchestration
• Automatic Discovery of Interpretable Planning Strategies
• Structure in Deep Reinforcement Learning: A Survey and Open Problems
• Counterfactual States for Atari Agents via Generative Deep Learning
• Goal-directed graph construction using reinforcement learning
• What Did You Think Would Happen? Explaining Agent Behaviour Through Intended Outcomes
• Domain-Level Explainability -- A Challenge for Creating Trust in Superhuman AI Strategies
• Towards Mixed Optimization for Reinforcement Learning with Program Synthesis
• Identifying Critical States by the Action-Based Variance of Expected Return
• Program Synthesis with Best-First Bottom-Up Search
• SDRL: Interpretable and Data-efficient Deep Reinforcement Learning Leveraging Symbolic Planning
• Evolutionary Selective Imitation: Interpretable Agents by Imitation Learning Without a Demonstrator
• Not All Features Are Equal: Feature Leveling Deep Neural Networks for Better Interpretation
• Worrisome Properties of Neural Network Controllers and Their Symbolic Representations
• Don't Forget Your Teacher: A Corrective Reinforcement Learning Framework
• Unsupervised Learning of Neurosymbolic Encoders
• Few-Shot Bayesian Imitation Learning with Logical Program Policies
• Deep Reinforcement Learning with Mixed Convolutional Network
• Achilles Heels for AGI/ASI via Decision Theoretic Adversaries
• Type-driven Neural Programming by Example