Worrisome Properties of Neural Network Controllers and Their Symbolic Representations

Abstract

We raise concerns about controllers' robustness in simple reinforcement learning benchmark problems. We focus on neural network controllers and their low neuron and symbolic abstractions. A typical controller reaching high mean return values still generates an abundance of persistent low-return solutions, which is a highly undesirable property, easily exploitable by an adversary. We find that the simpler controllers admit more persistent bad solutions. We provide an algorithm for a systematic robustness study and prove existence of persistent solutions and, in some cases, periodic orbits, using a computer-assisted proof methodology.

arXiv commentaccepted to ECAI23

References in corpus (16)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Continuous control with deep reinforcement learning
• Explaining and Harnessing Adversarial Examples
• Addressing Function Approximation Error in Actor-Critic Methods
• Soft Actor-Critic Algorithms and Applications
• DeepMind Control Suite
• Robust Adversarial Reinforcement Learning
• Discovering Symbolic Models from Deep Learning with Inductive Biases
• A Unified View of Piecewise Linear Neural Network Verification
• Forward-Mode Automatic Differentiation in Julia
• Programmatically Interpretable Reinforcement Learning
• Falsification of Cyber-Physical Systems Using Deep Reinforcement Learning
• On Robustness of Neural Ordinary Differential Equations
• Falsification of Cyber-Physical Systems with Robustness-Guided Black-Box Checking
• Learning to Synthesize Programs as Interpretable and Generalizable Policies
• Computer Validation of Neural Network Dynamics: A First Case Study