Behavior From the Void: Unsupervised Active Pre-Training

Abstract

We introduce a new unsupervised pre-training method for reinforcement learning called APT, which stands for Active Pre-Training. APT learns behaviors and representations by actively searching for novel states in reward-free environments. The key novel idea is to explore the environment by maximizing a non-parametric entropy computed in an abstract representation space, which avoids challenging density modeling and consequently allows our approach to scale much better in environments that have high-dimensional observations (e.g., image observations). We empirically evaluate APT by exposing task-specific reward after a long unsupervised pre-training phase. In Atari games, APT achieves human-level performance on 12 games and obtains highly competitive performance compared to canonical fully supervised RL algorithms. On DMControl suite, APT beats all baselines in terms of asymptotic performance and data efficiency and dramatically improves performance on tasks that are extremely difficult to train from scratch.

arXiv commentAdvances in Neural Information Processing Systems(NeurIPS), 2021 Spotlight

References in corpus (32)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Auto-Encoding Variational Bayes
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Spectral Normalization for Generative Adversarial Networks
• An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks
• Solving Rubik's Cube with a Robot Hand
• VIME: Variational Information Maximizing Exploration
• Rainbow: Combining Improvements in Deep Reinforcement Learning
• Model-Based Reinforcement Learning for Atari
• Count-Based Exploration with Neural Density Models
• Hindsight Experience Replay
• Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning
• Reinforcement Learning with Augmented Data
• Agent57: Outperforming the Atari Human Benchmark
• dm_control: Software and Tasks for Continuous Control
• Variational Intrinsic Control
• Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
• When to use parametric models in reinforcement learning?
• Provably Efficient Maximum Entropy Exploration
• Efficient Exploration via State Marginal Matching
• Never Give Up: Learning Directed Exploration Strategies
• Decoupling Representation Learning from Reinforcement Learning
• Planning to Explore via Self-Supervised World Models
• Unsupervised Control Through Non-Parametric Discriminative Rewards
• Reinforcement Learning with Prototypical Representations
• Never Stop Learning: The Effectiveness of Fine-Tuning in Robotic Reinforcement Learning
• Maximum Entropy Gain Exploration for Long Horizon Multi-goal Reinforcement Learning
• The Nearest Neighbor Information Estimator is Adaptively Near Minimax Rate-Optimal
• Fast Task Inference with Variational Intrinsic Successor Features
• APS: Active Pretraining with Successor Features
• Beyond Fine-Tuning: Transferring Behavior in Reinforcement Learning
• Novelty Search in Representational Space for Sample Efficient Exploration

Cited by in corpus (6)

• Deep Reinforcement Learning at the Edge of the Statistical Precipice
• State Entropy Maximization with Random Encoders for Efficient Exploration
• Dynamic Bottleneck for Robust Self-Supervised Exploration
• PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training
• Beyond Fine-Tuning: Transferring Behavior in Reinforcement Learning
• Pretrained Encoders are All You Need