Reinforcement Learning with Perturbed Rewards

Abstract

Recent studies have shown that reinforcement learning (RL) models are vulnerable in various noisy scenarios. For instance, the observed reward channel is often subject to noise in practice (e.g., when rewards are collected through sensors), and is therefore not credible. In addition, for applications such as robotics, a deep reinforcement learning (DRL) algorithm can be manipulated to produce arbitrary errors by receiving corrupted rewards. In this paper, we consider noisy RL problems with perturbed rewards, which can be approximated with a confusion matrix. We develop a robust RL framework that enables agents to learn in noisy environments where only perturbed rewards are observed. Our solution framework builds on existing RL/DRL algorithms and firstly addresses the biased noisy reward setting without any assumptions on the true distribution (e.g., zero-mean Gaussian noise as made in previous works). The core ideas of our solution include estimating a reward confusion matrix and defining a set of unbiased surrogate rewards. We prove the convergence and sample complexity of our approach. Extensive experiments on different DRL platforms show that trained policies based on our estimated surrogate reward can achieve higher expected rewards, and converge faster than existing baselines. For instance, the state-of-the-art PPO algorithm is able to obtain 84.6% and 80.8% improvements on average score for five Atari games, with error rates as 10% and 30% respectively.

arXiv commentAAAI 2020 (Spotlight)

References in corpus (15)

• Continuous control with deep reinforcement learning
• Playing Atari with Deep Reinforcement Learning
• Dueling Network Architectures for Deep Reinforcement Learning
• Training Convolutional Networks with Noisy Labels
• Robust Adversarial Reinforcement Learning
• Continuous Deep Q-Learning with Model-based Acceleration
• Distral: Robust Multitask Reinforcement Learning
• Delving into adversarial attacks on deep policies
• Using Trusted Data to Train Deep Networks on Labels Corrupted by Severe Noise
• Learning From Noisy Singly-labeled Data
• Tactics of Adversarial Attack on Deep Reinforcement Learning Agents
• Inverse Reward Design
• Adversary A3C for Robust Reinforcement Learning
• Reinforcement Learning under Model Mismatch
• Learning in the Presence of Corruption

Cited by in corpus (7)

• Deceptive Reinforcement Learning Under Adversarial Manipulations on Cost Signals
• A Reinforcement Learning Formulation of the Lyapunov Optimization: Application to Edge Computing Systems with Queue Stability
• One Solution is Not All You Need: Few-Shot Extrapolation via Structured MaxEnt RL
• Balance Between Efficient and Effective Learning: Dense2Sparse Reward Shaping for Robot Manipulation with Environment Uncertainty
• Manipulating Reinforcement Learning: Poisoning Attacks on Cost Signals
• Disturbing Reinforcement Learning Agents with Corrupted Rewards
• Variational Policy Search using Sparse Gaussian Process Priors for Learning Multimodal Optimal Actions