A Distance-based Anomaly Detection Framework for Deep Reinforcement Learning
arXiv:2109.09889
Abstract
In deep reinforcement learning (RL) systems, abnormal states pose significant risks by potentially triggering unpredictable behaviors and unsafe actions, thus impeding the deployment of RL systems in real-world scenarios. It is crucial for reliable decision-making systems to have the capability to cast an alert whenever they encounter unfamiliar observations that they are not equipped to handle. In this paper, we propose a novel Mahalanobis distance-based (MD) anomaly detection framework, called \textit{MDX}, for deep RL algorithms. MDX simultaneously addresses random, adversarial, and out-of-distribution (OOD) state outliers in both offline and online settings. It utilizes Mahalanobis distance within class-conditional distributions for each action and operates within a statistical hypothesis testing framework under the Gaussian assumption. We further extend it to robust and distribution-free versions by incorporating Robust MD and conformal inference techniques. Through extensive experiments on classical control environments, Atari games, and autonomous driving scenarios, we demonstrate the effectiveness of our MD-based detection framework. MDX offers a simple, unified, and practical anomaly detection tool for enhancing the safety and reliability of RL systems in real-world applications.
19 pages, 21 figures
References in corpus (9)
- Trust Region Policy Optimization
- High-Dimensional Continuous Control Using Generalized Advantage Estimation
- A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks
- Deep Reinforcement Learning that Matters
- A Simple Fix to Mahalanobis Distance for Improving Near-OOD Detection
- How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experiments
- Why is the Mahalanobis Distance Effective for Anomaly Detection?
- Measuring the Reliability of Reinforcement Learning Algorithms
- Evaluating the Performance of Reinforcement Learning Algorithms