Online Safety Assurance for Deep Reinforcement Learning

Abstract

Recently, deep learning has been successfully applied to a variety of networking problems. A fundamental challenge is that when the operational environment for a learning-augmented system differs from its training environment, such systems often make badly informed decisions, leading to bad performance. We argue that safely deploying learning-driven systems requires being able to determine, in real time, whether system behavior is coherent, for the purpose of defaulting to a reasonable heuristic when this is not so. We term this the online safety assurance problem (OSAP). We present three approaches to quantifying decision uncertainty that differ in terms of the signal used to infer uncertainty. We illustrate the usefulness of online safety assurance in the context of the proposed deep reinforcement learning (RL) approach to video streaming. While deep RL for video streaming bests other approaches when the operational and training environments match, it is dominated by simple heuristics when the two differ. Our preliminary findings suggest that transitioning to a default policy when decision uncertainty is detected is key to enjoying the performance benefits afforded by leveraging ML without compromising on safety.

arXiv comment8 pages, to appear in The 19th ACM Workshop on Hot Topics in Networks (HotNets 2020)

References in corpus (13)

• Trust Region Policy Optimization
• On Calibration of Modern Neural Networks
• Deep Learning for Anomaly Detection: A Survey
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• VIME: Variational Information Maximizing Exploration
• Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction
• Invariant Risk Minimization
• Exploration by Random Network Distillation
• A Study on Overfitting in Deep Reinforcement Learning
• Gotta Learn Fast: A New Benchmark for Generalization in RL
• Safe Policy Improvement by Minimizing Robust Baseline Regret
• Planning to Explore via Self-Supervised World Models
• Safer Classification by Synthesis

Cited by in corpus (2)

• A Survey on AI Assurance
• Out-of-Distribution Dynamics Detection: RL-Relevant Benchmarks and Results