Measuring the Reliability of Reinforcement Learning Algorithms

Abstract

Lack of reliability is a well-known issue for reinforcement learning (RL) algorithms. This problem has gained increasing attention in recent years, and efforts to improve it have grown substantially. To aid RL researchers and production users with the evaluation and improvement of reliability, we propose a set of metrics that quantitatively measure different aspects of reliability. In this work, we focus on variability and risk, both during training and after learning (on a fixed policy). We designed these metrics to be general-purpose, and we also designed complementary statistical tests to enable rigorous comparisons on these metrics. In this paper, we first describe the desired properties of the metrics and their design, the aspects of reliability that they measure, and their applicability to different scenarios. We then describe the statistical tests and make additional practical recommendations for reporting results. The metrics and accompanying statistical tools have been made available as an open-source library at https://github.com/google-research/rl-reliability-metrics. We apply our metrics to a set of common RL algorithms and environments, compare them, and analyze the results.

Accepted for publication at ICLR 2020 (spotlight)

References in corpus (9)

• Continuous control with deep reinforcement learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Rainbow: Combining Improvements in Deep Reinforcement Learning
• A Distributional Perspective on Reinforcement Learning
• Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control
• Dopamine: A Research Framework for Deep Reinforcement Learning
• Algorithms for CVaR Optimization in MDPs
• How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experiments
• Deterministic Implementations for Reproducibility in Deep Reinforcement Learning

Cited by in corpus (11)

• Deep Reinforcement Learning at the Edge of the Statistical Precipice
• CausalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learning
• AKF-SR: Adaptive Kalman Filtering-based Successor Representation
• HiER: Highlight Experience Replay for Boosting Off-Policy Reinforcement Learning Agents
• A Mapping of Assurance Techniques for Learning Enabled Autonomous Systems to the Systems Engineering Lifecycle
• Correlation-aware Cooperative Multigroup Broadcast 360° Video Delivery Network: A Hierarchical Deep Reinforcement Learning Approach
• A Distance-based Anomaly Detection Framework for Deep Reinforcement Learning
• Detecting Rewards Deterioration in Episodic Reinforcement Learning
• Can Reinforcement Learning for Continuous Control Generalize Across Physics Engines?
• Continuous Control With Ensemble Deep Deterministic Policy Gradients
• Is High Variance Unavoidable in RL? A Case Study in Continuous Control