Robust Adversarial Reinforcement Learning

Abstract

Deep neural networks coupled with fast simulation and improved computation have led to recent successes in the field of reinforcement learning (RL). However, most current RL-based approaches fail to generalize since: (a) the gap between simulation and real world is so large that policy-learning approaches fail to transfer; (b) even if policy learning is done in real world, the data scarcity leads to failed generalization from training to test scenarios (e.g., due to different friction or object masses). Inspired from H-infinity control methods, we note that both modeling errors and differences in training and test scenarios can be viewed as extra forces/disturbances in the system. This paper proposes the idea of robust adversarial reinforcement learning (RARL), where we train an agent to operate in the presence of a destabilizing adversary that applies disturbance forces to the system. The jointly trained adversary is reinforced -- that is, it learns an optimal destabilization policy. We formulate the policy learning as a zero-sum, minimax objective function. Extensive experiments in multiple environments (InvertedPendulum, HalfCheetah, Swimmer, Hopper and Walker2d) conclusively demonstrate that our method (a) improves training stability; (b) is robust to differences in training/test conditions; and c) outperform the baseline even in the absence of the adversary.

10 pages

References in corpus (6)

• Continuous control with deep reinforcement learning
• Neural Architecture Search with Reinforcement Learning
• Trust Region Policy Optimization
• Benchmarking Deep Reinforcement Learning for Continuous Control
• Transfer from Simulation to Real World through Learning Deep Inverse Dynamics Model
• EPOpt: Learning Robust Neural Network Policies Using Model Ensembles

Cited by in corpus (148)

• A Survey and Critique of Multiagent Deep Reinforcement Learning
• Solving Rubik's Cube with a Robot Hand
• How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned
• Reinforcement Learning for Selective Key Applications in Power Systems: Recent Advances and Future Challenges
• Learning Dexterous In-Hand Manipulation
• Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey
• A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
• Adversarial Policies: Attacking Deep Reinforcement Learning
• Fusion of Model-free Reinforcement Learning with Microgrid Control: Review and Vision
• Action Robust Reinforcement Learning and Applications in Continuous Control
• Assessing Transferability from Simulation to Reality for Reinforcement Learning
• Self-Supervised Policy Adaptation during Deployment
• GeneraLight: Improving Environment Generalization of Traffic Signal Control via Meta Reinforcement Learning
• Characterizing Attacks on Deep Reinforcement Learning
• Learning Scheduling Algorithms for Data Processing Clusters
• Robust Reinforcement Learning on State Observations with Learned Optimal Adversary
• Certified Adversarial Robustness for Deep Reinforcement Learning
• Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations
• Enforcing Policy Feasibility Constraints through Differentiable Projection for Energy Optimization
• Wasserstein Robust Reinforcement Learning
• Automatic Curriculum Learning through Value Disagreement
• Investigating Generalisation in Continuous Deep Reinforcement Learning
• Worst Cases Policy Gradients
• Robust Control for Dynamical Systems With Non-Gaussian Noise via Formal Abstractions
• If MaxEnt RL is the Answer, What is the Question?
• DeepRacer: Educational Autonomous Racing Platform for Experimentation with Sim2Real Reinforcement Learning
• CausalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learning
• A non-cooperative meta-modeling game for automated third-party calibrating, validating, and falsifying constitutive laws with parallelized adversarial attacks
• Maximum Entropy RL (Provably) Solves Some Robust RL Problems
• The Ingredients of Real-World Robotic Reinforcement Learning
• Learning to be Safe: Deep RL with a Safety Critic
• Towards Resilient Artificial Intelligence: Survey and Research Issues
• Context-aware Dynamics Model for Generalization in Model-Based Reinforcement Learning
• Analysing Deep Reinforcement Learning Agents Trained with Domain Randomisation
• Adversary A3C for Robust Reinforcement Learning
• Robust Model-based Reinforcement Learning for Autonomous Greenhouse Control
• The Faults in Our Pi Stars: Security Issues and Open Challenges in Deep Reinforcement Learning
• Generalization in Transfer Learning
• Locally Private Distributed Reinforcement Learning
• Adversarial Reinforced Instruction Attacker for Robust Vision-Language Navigation
• Saddle Point Optimization with Approximate Minimization Oracle and its Application to Robust Berthing Control
• Optimal Attacks on Reinforcement Learning Policies
• Distributionally Robust Reinforcement Learning
• Observational Overfitting in Reinforcement Learning
• CopyCAT: Taking Control of Neural Policies with Constant Attacks
• Deep Reinforcement Learning for Market Making Under a Hawkes Process-Based Limit Order Book Model
• Deep Reinforcement Learning with Robust and Smooth Policy
• Contextual Imagined Goals for Self-Supervised Robotic Learning
• Reinforcement Learning with Perturbed Rewards
• An Imitation from Observation Approach to Transfer Learning with Dynamics Mismatch
• Robust Deep Reinforcement Learning through Adversarial Loss
• Who Is the Strongest Enemy? Towards Optimal and Efficient Evasion Attacks in Deep RL
• Solving smooth min-min and min-max problems by mixed oracle algorithms
• How You Act Tells a Lot: Privacy-Leakage Attack on Deep Reinforcement Learning
• Vulnerability-Aware Poisoning Mechanism for Online RL with Unknown Dynamics
• Falsification-Based Robust Adversarial Reinforcement Learning
• Robust Reinforcement Learning using Adversarial Populations
• Provably Efficient Black-Box Action Poisoning Attacks Against Reinforcement Learning
• Zeroth-Order Algorithms for Smooth Saddle-Point Problems
• The limits of min-max optimization algorithms: convergence to spurious non-critical sets
• Adversarial Evaluation of Autonomous Vehicles in Lane-Change Scenarios
• Enforcing robust control guarantees within neural network policies
• PAC-Bayesian Bound for the Conditional Value at Risk
• Robust Reinforcement Learning using Least Squares Policy Iteration with Provable Performance Guarantees
• Robust Reinforcement Learning via Adversarial training with Langevin Dynamics
• Reinforcement Learning under Model Mismatch
• Risk Averse Robust Adversarial Reinforcement Learning
• SimGAN: Hybrid Simulator Identification for Domain Adaptation via Adversarial Reinforcement Learning
• Simulation to Scaled City: Zero-Shot Policy Transfer for Traffic Control via Autonomous Vehicles
• Robust Policy Gradient against Strong Data Corruption
• Risk Adversarial Learning System for Connected and Autonomous Vehicle Charging
• Improving Robustness via Risk Averse Distributional Reinforcement Learning
• Adversarial Attacks on Reinforcement Learning based Energy Management Systems of Extended Range Electric Delivery Vehicles
• One Solution is Not All You Need: Few-Shot Extrapolation via Structured MaxEnt RL
• AlphaGAN: Fully Differentiable Architecture Search for Generative Adversarial Networks
• Derivative-Free Policy Optimization for Linear Risk-Sensitive and Robust Control Design: Implicit Regularization and Sample Complexity
• Adversarial Reinforcement Learning for Observer Design in Autonomous Systems under Cyber Attacks
• $H_{\infty }$ Model-free Reinforcement Learning with Robust Stability Guarantee
• Adaptive extra-gradient methods for min-max optimization and games
• Robust Reinforcement Learning Under Minimax Regret for Green Security
• Evaluating the Robustness of Collaborative Agents
• Context-Aware Safe Reinforcement Learning for Non-Stationary Environments
• Improving Generalization of Reinforcement Learning with Minimax Distributional Soft Actor-Critic
• Solving Dual Sourcing Problems with Supply Mode Dependent Failure Rates
• Sample Complexity of Robust Reinforcement Learning with a Generative Model
• Saddle Point Optimization with Approximate Minimization Oracle
• Risk-Averse Bayes-Adaptive Reinforcement Learning
• Exploration-efficient Deep Reinforcement Learning with Demonstration Guidance for Robot Control
• Two-stage Deep Reinforcement Learning for Inverter-based Volt-VAR Control in Active Distribution Networks
• Twice regularized MDPs and the equivalence between robustness and regularization
• Enhanced Adversarial Strategically-Timed Attacks against Deep Reinforcement Learning
• Online Robust Reinforcement Learning with Model Uncertainty
• ADAIL: Adaptive Adversarial Imitation Learning
• MRAC-RL: A Framework for On-Line Policy Adaptation Under Parametric Model Uncertainty
• Analysis and Improvement of Adversarial Training in DQN Agents With Adversarially-Guided Exploration (AGE)
• Learning Agile Locomotion via Adversarial Training
• Adversarial joint attacks on legged robots
• Understanding the Safety Requirements for Learning-based Power Systems Operations
• CROP: Towards Distributional-Shift Robust Reinforcement Learning using Compact Reshaped Observation Processing
• No-Regret Reinforcement Learning with Heavy-Tailed Rewards
• Unsupervised Visual Attention and Invariance for Reinforcement Learning
• Optimistic Policy Optimization is Provably Efficient in Non-stationary MDPs
• Certifiable Robustness to Adversarial State Uncertainty in Deep Reinforcement Learning
• Combining Pessimism with Optimism for Robust and Efficient Model-Based Deep Reinforcement Learning
• Robustness to Adversarial Attacks in Learning-Enabled Controllers
• SENTINEL: Taming Uncertainty with Ensemble-based Distributional Reinforcement Learning
• Multi-Task Federated Reinforcement Learning with Adversaries
• Robust Inverse Reinforcement Learning under Transition Dynamics Mismatch
• CARL: A Benchmark for Contextual and Adaptive Reinforcement Learning
• Disturbing Reinforcement Learning Agents with Corrupted Rewards
• FormulaZero: Distributionally Robust Online Adaptation via Offline Population Synthesis
• Towards Robust Off-Policy Evaluation via Human Inputs
• Survival of the Fittest: Evolutionary Adaptation of Policies for Environmental Shifts
• Adversarial Active Exploration for Inverse Dynamics Model Learning
• Fault-Aware Robust Control via Adversarial Reinforcement Learning
• Learning Robust Options by Conditional Value at Risk Optimization
• The Principle of Unchanged Optimality in Reinforcement Learning Generalization
• Finite-Time Complexity of Online Primal-Dual Natural Actor-Critic Algorithm for Constrained Markov Decision Processes
• Dual-Arm Adversarial Robot Learning
• A Generative Model based Adversarial Security of Deep Learning and Linear Classifier Models
• Worrisome Properties of Neural Network Controllers and Their Symbolic Representations
• An Active Learning Framework for Efficient Robust Policy Search
• Comparing Task Simplifications to Learn Closed-Loop Object Picking Using Deep Reinforcement Learning
• Minimalistic Attacks: How Little it Takes to Fool a Deep Reinforcement Learning Policy
• Robust Risk-Sensitive Reinforcement Learning Agents for Trading Markets
• Improved Robustness and Safety for Autonomous Vehicle Control with Adversarial Reinforcement Learning
• Scalable Safety-Critical Policy Evaluation with Accelerated Rare Event Sampling
• Automatic Risk Adaptation in Distributional Reinforcement Learning
• Zero-Shot Adaptation for mmWave Beam-Tracking on Overhead Messenger Wires through Robust Adversarial Reinforcement Learning
• Efficient Policy Learning for Non-Stationary MDPs under Adversarial Manipulation
• A Study on the Uncertainty of Convolutional Layers in Deep Neural Networks
• Adversary Agnostic Robust Deep Reinforcement Learning
• Representation of Reinforcement Learning Policies in Reproducing Kernel Hilbert Spaces
• A Natural Actor-Critic Algorithm with Downside Risk Constraints
• L2E: Learning to Exploit Your Opponent
• Adversarial Skill Learning for Robust Manipulation
• Learning Robust Controllers Via Probabilistic Model-Based Policy Search
• Robustifying Reinforcement Learning Policies with ${\mathcal{L}}_1$ Adaptive Control
• Contributions to Large Scale Bayesian Inference and Adversarial Machine Learning
• Constrained Policy Gradient Method for Safe and Fast Reinforcement Learning: a Neural Tangent Kernel Based Approach
• MBDP: A Model-based Approach to Achieve both Robustness and Sample Efficiency via Double Dropout Planning
• Model-Free $\mu$ Synthesis via Adversarial Reinforcement Learning
• A Game Theoretic Analysis of LQG Control under Adversarial Attack
• Motion Generation Considering Situation with Conditional Generative Adversarial Networks for Throwing Robots
• Reward-Free Attacks in Multi-Agent Reinforcement Learning
• Data-Driven Pole Placement in LMI Regions with Robustness Constraints
• Policy Transfer across Visual and Dynamics Domain Gaps via Iterative Grounding
• CoachNet: An Adversarial Sampling Approach for Reinforcement Learning