Verifiable Reinforcement Learning via Policy Extraction

Abstract

While deep reinforcement learning has successfully solved many challenging control tasks, its real-world applicability has been limited by the inability to ensure the safety of learned policies. We propose an approach to verifiable reinforcement learning by training decision tree policies, which can represent complex policies (since they are nonparametric), yet can be efficiently verified using existing techniques (since they are highly structured). The challenge is that decision tree policies are difficult to train. We propose VIPER, an algorithm that combines ideas from model compression and imitation learning to learn decision tree policies guided by a DNN policy (called the oracle) and its Q-function, and show that it substantially outperforms two baselines. We use VIPER to (i) learn a provably robust decision tree policy for a variant of Atari Pong with a symbolic state space, (ii) learn a decision tree policy for a toy game based on Pong that provably never loses, and (iii) learn a provably stable decision tree policy for cart-pole. In each case, the decision tree policy achieves performance equal to that of the original DNN policy.

References in corpus (4)

• Distilling the Knowledge in a Neural Network
• Programmatically Interpretable Reinforcement Learning
• Safe Exploration in Markov Decision Processes
• GENESIM: genetic extraction of a single, interpretable model

Cited by in corpus (31)

• Explainable Deep Reinforcement Learning: State of the Art and Challenges
• Scalable agent alignment via reward modeling: a research direction
• Reinforcement Learning in Healthcare: A Survey
• Interpreting Deep Learning-Based Networking Systems
• Adversarial Policies: Attacking Deep Reinforcement Learning
• Neurosymbolic Reinforcement Learning and Planning: A Survey
• Robustness Verification of Tree-based Models
• Formal Verification of Input-Output Mappings of Tree Ensembles
• MoËT: Mixture of Expert Trees and its Application to Verifiable Reinforcement Learning
• Towards Interpretable-AI Policies Induction using Evolutionary Nonlinear Decision Trees for Discrete Action Systems
• Automatic Discovery of Interpretable Planning Strategies
• MurTree: Optimal Classification Trees via Dynamic Programming and Search
• Proposed Guidelines for the Responsible Use of Explainable Machine Learning
• Learning Interpretable Models with Causal Guarantees
• Learning to Synthesize Programs as Interpretable and Generalizable Policies
• Imitation-Projected Programmatic Reinforcement Learning
• How to Control Hydrodynamic Force on Fluidic Pinball via Deep Reinforcement Learning
• Optimal Decision Tree Policies for Markov Decision Processes
• Feature-Based Interpretable Reinforcement Learning based on State-Transition Models
• Interpretable Modeling of Deep Reinforcement Learning Driven Scheduling
• Identification of Unexpected Decisions in Partially Observable Monte-Carlo Planning: a Rule-Based Approach
• Formal Language Constraints for Markov Decision Processes
• Towards Mixed Optimization for Reinforcement Learning with Program Synthesis
• Program Synthesis with Best-First Bottom-Up Search
• Program Synthesis Guided Reinforcement Learning for Partially Observed Environments
• Hybrid system identification using switching density networks
• Representation of Reinforcement Learning Policies in Reproducing Kernel Hilbert Spaces
• Designing Interpretable Approximations to Deep Reinforcement Learning
• Programming with Neural Surrogates of Programs
• Optimal Interpretability-Performance Trade-off of Classification Trees with Black-Box Reinforcement Learning
• Synthesizing Machine Learning Programs with PAC Guarantees via Statistical Sketching