Falsification-Based Robust Adversarial Reinforcement Learning

Abstract

Reinforcement learning (RL) has achieved enormous progress in solving various sequential decision-making problems, such as control tasks in robotics. Since policies are overfitted to training environments, RL methods have often failed to be generalized to safety-critical test scenarios. Robust adversarial RL (RARL) was previously proposed to train an adversarial network that applies disturbances to a system, which improves the robustness in test scenarios. However, an issue of neural network-based adversaries is that integrating system requirements without handcrafting sophisticated reward signals are difficult. Safety falsification methods allow one to find a set of initial conditions and an input sequence, such that the system violates a given property formulated in temporal logic. In this paper, we propose falsification-based RARL (FRARL): this is the first generic framework for integrating temporal logic falsification in adversarial learning to improve policy robustness. By applying our falsification method, we do not need to construct an extra reward function for the adversary. Moreover, we evaluate our approach on a braking assistance system and an adaptive cruise control system of autonomous vehicles. Our experimental results demonstrate that policies trained with a falsification-based adversary generalize better and show less violation of the safety specification in test scenarios than those trained without an adversary or with an adversarial network.

arXiv comment8 pages, 4 figures

References in corpus (10)

• Adam: A Method for Stochastic Optimization
• Explaining and Harnessing Adversarial Examples
• Congested Traffic States in Empirical Observations and Microscopic Simulations
• A Brief Survey of Deep Reinforcement Learning
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• Robust Adversarial Reinforcement Learning
• Delving into adversarial attacks on deep policies
• Robust Deep Reinforcement Learning with Adversarial Attacks
• Adversarial Attacks on Neural Network Policies
• From Specifications to Behavior: Maneuver Verification in a Semantic State Space

Cited by in corpus (3)

• A Survey of Reinforcement Learning for Optimization in Automation
• Ablation Study of How Run Time Assurance Impacts the Training and Performance of Reinforcement Learning Agents
• Synthesis-guided Adversarial Scenario Generation for Gray-box Feedback Control Systems with Sensing Imperfections