EPOpt: Learning Robust Neural Network Policies Using Model Ensembles

Abstract

Sample complexity and safety are major challenges when learning policies with reinforcement learning for real-world tasks, especially when the policies are represented using rich function approximators like deep neural networks. Model-based methods where the real-world target domain is approximated using a simulated source domain provide an avenue to tackle the above challenges by augmenting real data with simulated data. However, discrepancies between the simulated source domain and the target domain pose a challenge for simulated training. We introduce the EPOpt algorithm, which uses an ensemble of simulated source domains and a form of adversarial training to learn policies that are robust and generalize to a broad range of possible target domains, including unmodeled effects. Further, the probability distribution over source domains in the ensemble can be adapted using data from target domain and approximate Bayesian methods, to progressively make it a better approximation. Thus, learning on a model ensemble, along with source domain adaptation, provides the benefit of both robustness and learning/adaptation.

Accepted for publication at the International Conference on Learning Representations (ICLR) 2017. Supplementary video: https://youtu.be/w1YJ9vwaoto

References in corpus (4)

• Benchmarking Deep Reinforcement Learning for Continuous Control
• Model-Based Bayesian Exploration
• Learning Parameterized Skills
• Agnostic System Identification for Model-Based Reinforcement Learning

Cited by in corpus (65)

• DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills
• Robust Adversarial Reinforcement Learning
• Graph networks as learnable physics engines for inference and control
• Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World
• Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control
• Transfer Learning in Deep Reinforcement Learning: A Survey
• Reinforcement Learning for Angle-Only Intercept Guidance of Maneuvering Targets
• Robust Deep Reinforcement Learning with Adversarial Attacks
• Adaptive Guidance and Integrated Navigation with Reinforcement Meta-Learning
• Assessing Generalization in Deep Reinforcement Learning
• Sim-to-Real: Learning Agile Locomotion For Quadruped Robots
• Self-Supervised Policy Adaptation during Deployment
• Six Degree-of-Freedom Body-Fixed Hovering over Unmapped Asteroids via LIDAR Altimetry and Reinforcement Meta-Learning
• Policy Transfer with Strategy Optimization
• GeneraLight: Improving Environment Generalization of Traffic Signal Control via Meta Reinforcement Learning
• Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradients
• Preparing for the Unknown: Learning a Universal Policy with Online System Identification
• DeepRacer: Educational Autonomous Racing Platform for Experimentation with Sim2Real Reinforcement Learning
• Maximum Entropy RL (Provably) Solves Some Robust RL Problems
• Multi-Task Reinforcement Learning with Context-based Representations
• The Faults in Our Pi Stars: Security Issues and Open Challenges in Deep Reinforcement Learning
• Mutual Alignment Transfer Learning
• Robust Model-based Reinforcement Learning for Autonomous Greenhouse Control
• Generalization in Transfer Learning
• Locally Private Distributed Reinforcement Learning
• Deep Robust Kalman Filter
• Distributionally Robust Reinforcement Learning
• Dexterous Manipulation with Deep Reinforcement Learning: Efficient, General, and Low-Cost
• Situational Fusion of Visual Representation for Visual Navigation
• Model-based Adversarial Meta-Reinforcement Learning
• How You Act Tells a Lot: Privacy-Leakage Attack on Deep Reinforcement Learning
• Adversarial Evaluation of Autonomous Vehicles in Lane-Change Scenarios
• Robust Reinforcement Learning using Adversarial Populations
• Generalization through Simulation: Integrating Simulated and Real Data into Deep Reinforcement Learning for Vision-Based Autonomous Flight
• Robust Reinforcement Learning with Wasserstein Constraint
• SimGAN: Hybrid Simulator Identification for Domain Adaptation via Adversarial Reinforcement Learning
• Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifiers
• Map-based Multi-Policy Reinforcement Learning: Enhancing Adaptability of Robots by Deep Reinforcement Learning
• One Solution is Not All You Need: Few-Shot Extrapolation via Structured MaxEnt RL
• Improving Generalization of Reinforcement Learning with Minimax Distributional Soft Actor-Critic
• Attention-Privileged Reinforcement Learning
• Two-stage Deep Reinforcement Learning for Inverter-based Volt-VAR Control in Active Distribution Networks
• MRAC-RL: A Framework for On-Line Policy Adaptation Under Parametric Model Uncertainty
• ADAIL: Adaptive Adversarial Imitation Learning
• Unsupervised Visual Attention and Invariance for Reinforcement Learning
• Machine Learning for Mechanical Ventilation Control
• Meta-Model-Based Meta-Policy Optimization
• TuneNet: One-Shot Residual Tuning for System Identification and Sim-to-Real Robot Task Transfer
• Block Contextual MDPs for Continual Learning
• VMAV-C: A Deep Attention-based Reinforcement Learning Algorithm for Model-based Control
• On Effective Scheduling of Model-based Reinforcement Learning
• Learning Powerful Policies by Using Consistent Dynamics Model
• Sparse Attention Guided Dynamic Value Estimation for Single-Task Multi-Scene Reinforcement Learning
• Unifying Ensemble Methods for Q-learning via Social Choice Theory
• Dynamic Value Estimation for Single-Task Multi-Scene Reinforcement Learning
• Deep Model-Based Reinforcement Learning via Estimated Uncertainty and Conservative Policy Optimization
• Zero-shot Deep Reinforcement Learning Driving Policy Transfer for Autonomous Vehicles based on Robust Control
• Improved Robustness and Safety for Autonomous Vehicle Control with Adversarial Reinforcement Learning
• Quasi-Newton Trust Region Policy Optimization
• On the Expressivity of Neural Networks for Deep Reinforcement Learning
• Motion Generation Considering Situation with Conditional Generative Adversarial Networks for Throwing Robots
• MBDP: A Model-based Approach to Achieve both Robustness and Sample Efficiency via Double Dropout Planning
• Learning to Actively Learn: A Robust Approach
• Zero-shot generalization using cascaded system-representations
• Planning with Exploration: Addressing Dynamics Bottleneck in Model-based Reinforcement Learning