Worrisome Properties of Neural Network Controllers and Their Symbolic Representations

Abstract

We raise concerns about controllers' robustness in simple reinforcement learning benchmark problems. We focus on neural network controllers and their low neuron and symbolic abstractions. A typical controller reaching high mean return values still generates an abundance of persistent low-return solutions, which is a highly undesirable property, easily exploitable by an adversary. We find that the simpler controllers admit more persistent bad solutions. We provide an algorithm for a systematic robustness study and prove existence of persistent solutions and, in some cases, periodic orbits, using a computer-assisted proof methodology.

accepted to ECAI23

References in corpus (7)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Explaining and Harnessing Adversarial Examples
• DeepMind Control Suite
• Robust Adversarial Reinforcement Learning
• Forward-Mode Automatic Differentiation in Julia
• Learning to Synthesize Programs as Interpretable and Generalizable Policies
• Computer Validation of Neural Network Dynamics: A First Case Study