Reinforcement Learning with Prototypical Representations

Abstract

Learning effective representations in image-based environments is crucial for sample efficient Reinforcement Learning (RL). Unfortunately, in RL, representation learning is confounded with the exploratory experience of the agent -- learning a useful representation requires diverse data, while effective exploration is only possible with coherent representations. Furthermore, we would like to learn representations that not only generalize across tasks but also accelerate downstream exploration for efficient task-specific training. To address these challenges we propose Proto-RL, a self-supervised framework that ties representation learning with exploration through prototypical representations. These prototypes simultaneously serve as a summarization of the exploratory experience of an agent as well as a basis for representing observations. We pre-train these task-agnostic representations and prototypes on environments without downstream task information. This enables state-of-the-art downstream policy learning on a set of difficult continuous control tasks.

References in corpus (22)

• Adam: A Method for Stochastic Optimization
• Continuous control with deep reinforcement learning
• A Simple Framework for Contrastive Learning of Visual Representations
• Playing Atari with Deep Reinforcement Learning
• Bootstrap your own latent: A new approach to self-supervised Learning
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• Soft Actor-Critic Algorithms and Applications
• DeepMind Control Suite
• Count-Based Exploration with Neural Density Models
• Hindsight Experience Replay
• Exploration by Random Network Distillation
• Dream to Control: Learning Behaviors by Latent Imagination
• Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
• Efficient Exploration via State Marginal Matching
• Learning Predictive Representations for Deformable Objects Using Contrastive Estimation
• Decoupling Representation Learning from Reinforcement Learning
• Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model
• Planning to Explore via Self-Supervised World Models
• Data-Efficient Reinforcement Learning with Self-Predictive Representations
• Visual Imitation Made Easy
• Robust Policies via Mid-Level Visual Representations: An Experimental Study in Manipulation and Navigation
• Learning Visual Robotic Control Efficiently with Contrastive Pre-training and Data Augmentation

Cited by in corpus (11)

• Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain
• Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning
• Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentation
• Structure in Deep Reinforcement Learning: A Survey and Open Problems
• MADE: Exploration via Maximizing Deviation from Explored Regions
• Behavior From the Void: Unsupervised Active Pre-Training
• Learning Markov State Abstractions for Deep Reinforcement Learning
• Learning Temporally-Consistent Representations for Data-Efficient Reinforcement Learning
• Learning Representations for Pixel-based Control: What Matters and Why?
• Stein Variational Goal Generation for adaptive Exploration in Multi-Goal Reinforcement Learning
• Unsupervised Skill-Discovery and Skill-Learning in Minecraft