Invariant Policy Optimization: Towards Stronger Generalization in Reinforcement Learning

Abstract

A fundamental challenge in reinforcement learning is to learn policies that generalize beyond the operating domains experienced during training. In this paper, we approach this challenge through the following invariance principle: an agent must find a representation such that there exists an action-predictor built on top of this representation that is simultaneously optimal across all training domains. Intuitively, the resulting invariant policy enhances generalization by finding causes of successful actions. We propose a novel learning algorithm, Invariant Policy Optimization (IPO), that implements this principle and learns an invariant policy during training. We compare our approach with standard policy gradient methods and demonstrate significant improvements in generalization performance on unseen domains for linear quadratic regulator and grid-world problems, and an example where a robot must learn to open doors with varying physical properties.

arXiv comment16 pages, 4 figures

References in corpus (18)

• Adam: A Method for Stochastic Optimization
• Playing Atari with Deep Reinforcement Learning
• Solving Rubik's Cube with a Robot Hand
• Certifying Some Distributional Robustness with Principled Adversarial Training
• Invariant Risk Minimization
• Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulator
• Out-of-Distribution Generalization via Risk Extrapolation (REx)
• Leveraging Procedural Generation to Benchmark Reinforcement Learning
• Sim-to-Real: Learning Agile Locomotion For Quadruped Robots
• Gotta Learn Fast: A New Benchmark for Generalization in RL
• Causal Reasoning from Meta-reinforcement Learning
• Causal Induction from Visual Observations for Goal Directed Tasks
• InfoBot: Transfer and Exploration via the Information Bottleneck
• Unshuffling Data for Improved Generalization
• Invariant Causal Prediction for Block MDPs
• Invariant Risk Minimization Games
• DoorGym: A Scalable Door Opening Environment And Baseline Agent
• Observational Overfitting in Reinforcement Learning

Cited by in corpus (8)

• A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
• Automatic Data Augmentation for Generalization in Deep Reinforcement Learning
• Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning
• Measuring Visual Generalization in Continuous Control from Pixels
• Model-Based Domain Generalization
• When Is Generalizable Reinforcement Learning Tractable?
• Task-relevant Representation Learning for Networked Robotic Perception
• Decoupling Value and Policy for Generalization in Reinforcement Learning