Beyond Fine-Tuning: Transferring Behavior in Reinforcement Learning
arXiv:2102.13515
Abstract
Designing agents that acquire knowledge autonomously and use it to solve new tasks efficiently is an important challenge in reinforcement learning. Knowledge acquired during an unsupervised pre-training phase is often transferred by fine-tuning neural network weights once rewards are exposed, as is common practice in supervised domains. Given the nature of the reinforcement learning problem, we argue that standard fine-tuning strategies alone are not enough for efficient transfer in challenging domains. We introduce Behavior Transfer (BT), a technique that leverages pre-trained policies for exploration and that is complementary to transferring neural network weights. Our experiments show that, when combined with large-scale pre-training in the absence of rewards, existing intrinsic motivation objectives can lead to the emergence of complex behaviors. These pre-trained policies can then be leveraged by BT to discover better solutions than without pre-training, and combining BT with standard fine-tuning strategies results in additional benefits. The largest gains are generally observed in domains requiring structured exploration, including settings where the behavior of the pre-trained policies is misaligned with the downstream task.
References in corpus (24)
- Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
- DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
- How transferable are features in deep neural networks?
- Bootstrap your own latent: A new approach to self-supervised Learning
- Language Models are Few-Shot Learners
- Dueling Network Architectures for Deep Reinforcement Learning
- Scaling Laws for Neural Language Models
- Momentum Contrast for Unsupervised Visual Representation Learning
- Data-Efficient Image Recognition with Contrastive Predictive Coding
- Distributed Prioritized Experience Replay
- Recurrent World Models Facilitate Policy Evolution
- Large-Scale Study of Curiosity-Driven Learning
- Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning
- Exploration by Random Network Distillation
- Agent57: Outperforming the Atari Human Benchmark
- Variational Intrinsic Control
- Provably Efficient Maximum Entropy Exploration
- Efficient Exploration via State Marginal Matching
- Never Give Up: Learning Directed Exploration Strategies
- Decoupling Representation Learning from Reinforcement Learning
- Unsupervised Meta-Learning for Reinforcement Learning
- Unsupervised Curricula for Visual Meta-Reinforcement Learning
- Behavior From the Void: Unsupervised Active Pre-Training
- When Simple Exploration is Sample Efficient: Identifying Sufficient Conditions for Random Exploration to Yield PAC RL Algorithms