Decoupling Value and Policy for Generalization in Reinforcement Learning
arXiv:2102.10330
Abstract
Standard deep reinforcement learning algorithms use a shared representation for the policy and value function, especially when training directly from images. However, we argue that more information is needed to accurately estimate the value function than to learn the optimal policy. Consequently, the use of a shared representation for the policy and value function can lead to overfitting. To alleviate this problem, we propose two approaches which are combined to create IDAAC: Invariant Decoupled Advantage Actor-Critic. First, IDAAC decouples the optimization of the policy and value function, using separate networks to model them. Second, it introduces an auxiliary loss which encourages the representation to be invariant to task-irrelevant properties of the environment. IDAAC shows good generalization to unseen environments, achieving a new state-of-the-art on the Procgen benchmark and outperforming popular methods on DeepMind Control tasks with distractors. Our implementation is available at https://github.com/rraileanu/idaac.
References in corpus (33)
- Trust Region Policy Optimization
- The Kinetics Human Action Video Dataset
- Dueling Network Architectures for Deep Reinforcement Learning
- IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
- DeepMind Control Suite
- Distributed Distributional Deterministic Policy Gradients
- Invariant Risk Minimization
- Reinforcement Learning with Augmented Data
- Quantifying Generalization in Reinforcement Learning
- Leveraging Procedural Generation to Benchmark Reinforcement Learning
- Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
- Illuminating Generalization in Deep Reinforcement Learning through Procedural Level Generation
- Assessing Generalization in Deep Reinforcement Learning
- Generalization and Regularization in DQN
- A Dissection of Overfitting and Generalization in Continuous Reinforcement Learning
- Gotta Learn Fast: A New Benchmark for Generalization in RL
- Learning Invariant Representations for Reinforcement Learning without Reconstruction
- Decoupling Representation Learning from Reinforcement Learning
- Transfer Learning for Related Reinforcement Learning Tasks via Image-to-Image Translation
- Generalization in Reinforcement Learning with Selective Noise Injection and Information Bottleneck
- Automatic Data Augmentation for Generalization in Deep Reinforcement Learning
- Phasic Policy Gradient
- The NetHack Learning Environment
- Improving Generalization in Reinforcement Learning with Mixture Regularization
- Invariant Causal Prediction for Block MDPs
- RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environments
- Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning
- Invariant Policy Optimization: Towards Stronger Generalization in Reinforcement Learning
- Interference and Generalization in Temporal Difference Learning
- Rotation, Translation, and Cropping for Zero-Shot Generalization
- Reinforcement Learning Generalization with Surprise Minimization
- Measuring Visual Generalization in Continuous Control from Pixels
- Prioritized Level Replay