Pretraining Representations for Data-Efficient Reinforcement Learning
arXiv:2106.04799
Abstract
Data efficiency is a key challenge for deep reinforcement learning. We address this problem by using unlabeled data to pretrain an encoder which is then finetuned on a small amount of task-specific data. To encourage learning representations which capture diverse aspects of the underlying MDP, we employ a combination of latent dynamics modelling and unsupervised goal-conditioned RL. When limited to 100k steps of interaction on Atari games (equivalent to two hours of human experience), our approach significantly surpasses prior work combining offline representation pretraining with task-specific finetuning, and compares favourably with other pretraining methods that require orders of magnitude more data. Our approach shows particular promise when combined with larger models as well as more diverse, task-aligned observational data -- approaching human-level performance and data-efficiency on Atari in our best setting. We provide code associated with this work at https://github.com/mila-iqia/SGI.
References in corpus (22)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Language Models are Few-Shot Learners
- FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
- Dueling Network Architectures for Deep Reinforcement Learning
- Learning deep representations by mutual information estimation and maximization
- Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
- Scaling Laws for Neural Language Models
- Continual Learning Through Synaptic Intelligence
- Data-Efficient Image Recognition with Contrastive Predictive Coding
- Rainbow: Combining Improvements in Deep Reinforcement Learning
- State Representation Learning for Control: An Overview
- A Study on Overfitting in Deep Reinforcement Learning
- Dopamine: A Research Framework for Deep Reinforcement Learning
- When to use parametric models in reinforcement learning?
- Decoupling Representation Learning from Reinforcement Learning
- Rethinking the Hyperparameters for Fine-tuning
- rlpyt: A Research Code Base for Deep Reinforcement Learning in PyTorch
- An empirical investigation of the challenges of real-world reinforcement learning
- Neural Predictive Belief Representations
- Scaling Laws for Transfer
- Muesli: Combining Improvements in Policy Optimization