Adaptive Reward-Poisoning Attacks against Reinforcement Learning

Abstract

In reward-poisoning attacks against reinforcement learning (RL), an attacker can perturb the environment reward $r_t$ into $r_t+{\delta }_t$ at each step, with the goal of forcing the RL agent to learn a nefarious policy. We categorize such attacks by the infinity-norm constraint on ${\delta }_t$: We provide a lower threshold below which reward-poisoning attack is infeasible and RL is certified to be safe; we provide a corresponding upper threshold above which the attack is feasible. Feasible attacks can be further categorized as non-adaptive where ${\delta }_t$ depends only on $(s_t,a_t,s_{t+1})$, or adaptive where ${\delta }_t$ depends further on the RL agent's learning process at time $t$. Non-adaptive attacks have been the focus of prior works. However, we show that under mild conditions, adaptive attacks can achieve the nefarious policy in steps polynomial in state-space size $|S|$, whereas non-adaptive attacks require exponential steps. We provide a constructive proof that a Fast Adaptive Attack strategy achieves the polynomial rate. Finally, we show that empirically an attacker can find effective reward-poisoning attacks using state-of-the-art deep RL techniques.

References in corpus (4)

• Delving into adversarial attacks on deep policies
• Policy Poisoning in Batch Reinforcement Learning and Control
• Data Poisoning Attacks on Stochastic Bandits
• Deceptive Reinforcement Learning Under Adversarial Manipulations on Cost Signals

Cited by in corpus (10)

• Building Privacy-Preserving and Secure Geospatial Artificial Intelligence Foundation Models
• Vulnerability-Aware Poisoning Mechanism for Online RL with Unknown Dynamics
• Provably Efficient Black-Box Action Poisoning Attacks Against Reinforcement Learning
• Robust Policy Gradient against Strong Data Corruption
• Defense Against Reward Poisoning Attacks in Reinforcement Learning
• Using Machine Teaching to Investigate Human Assumptions when Teaching Reinforcement Learners
• Learning-based attacks in Cyber-Physical Systems: Exploration, Detection, and Control Cost trade-offs
• The Sample Complexity of Teaching-by-Reinforcement on Q-Learning
• Sequential Attacks on Kalman Filter-based Forward Collision Warning Systems
• Deceptive Kernel Function on Observations of Discrete POMDP