Generalization in Reinforcement Learning with Selective Noise Injection and Information Bottleneck

Abstract

The ability for policies to generalize to new environments is key to the broad application of RL agents. A promising approach to prevent an agent's policy from overfitting to a limited set of training environments is to apply regularization techniques originally developed for supervised learning. However, there are stark differences between supervised learning and RL. We discuss those differences and propose modifications to existing regularization techniques in order to better adapt them to RL. In particular, we focus on regularization techniques relying on the injection of noise into the learned function, a family that includes some of the most widely used approaches such as Dropout and Batch Normalization. To adapt them to RL, we propose Selective Noise Injection (SNI), which maintains the regularizing effect the injected noise has, while mitigating the adverse effects it has on the gradient quality. Furthermore, we demonstrate that the Information Bottleneck (IB) is a particularly well suited regularization technique for RL as it is effective in the low-data regime encountered early on in training RL agents. Combining the IB with SNI, we significantly outperform current state of the art results, including on the recently proposed generalization benchmark Coinrun.

Published at Neurips 2019

Cited by in corpus (33)

• A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
• Automatic Data Augmentation for Generalization in Deep Reinforcement Learning
• Domain Generalization with MixStyle
• Variational Information Bottleneck for Effective Low-Resource Fine-Tuning
• Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning
• Predictive Information Accelerates Learning in RL
• Observational Overfitting in Reinforcement Learning
• MixStyle Neural Networks for Domain Generalization and Adaptation
• Randomized Entity-wise Factorization for Multi-Agent Reinforcement Learning
• SECANT: Self-Expert Cloning for Zero-Shot Generalization of Visual Policies
• Robust Predictable Control
• Adversarially Guided Actor-Critic
• Demystifying Deep Learning in Predictive Spatio-Temporal Analytics: An Information-Theoretic Framework
• Transient Non-Stationarity and Generalisation in Deep Reinforcement Learning
• Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observability
• One Solution is Not All You Need: Few-Shot Extrapolation via Structured MaxEnt RL
• Rotation, Translation, and Cropping for Zero-Shot Generalization
• Drop-Bottleneck: Learning Discrete Compressed Representation for Noise-Robust Exploration
• Procedural Generalization by Planning with Self-Supervised World Models
• Decoupling Value and Policy for Generalization in Reinforcement Learning
• Prioritized Level Replay
• Graph Information Bottleneck for Subgraph Recognition
• Improving Generalization in Meta-RL with Imaginary Tasks from Latent Dynamics Mixture
• Unlocking Pixels for Reinforcement Learning via Implicit Attention
• Off-Policy Meta-Reinforcement Learning Based on Feature Embedding Spaces
• Dynamic Value Estimation for Single-Task Multi-Scene Reinforcement Learning
• Sparse Attention Guided Dynamic Value Estimation for Single-Task Multi-Scene Reinforcement Learning
• Instance based Generalization in Reinforcement Learning
• Enhanced Scene Specificity with Sparse Dynamic Value Estimation
• Active Reinforcement Learning over MDPs
• Learning Transferable Concepts in Deep Reinforcement Learning
• Learning Robust Controllers Via Probabilistic Model-Based Policy Search
• Conditioning of Reinforcement Learning Agents and its Policy Regularization Application