Dynamics-aware Embeddings

Abstract

In this paper we consider self-supervised representation learning to improve sample efficiency in reinforcement learning (RL). We propose a forward prediction objective for simultaneously learning embeddings of states and action sequences. These embeddings capture the structure of the environment's dynamics, enabling efficient policy learning. We demonstrate that our action embeddings alone improve the sample efficiency and peak performance of model-free RL on control from low-dimensional states. By combining state and action embeddings, we achieve efficient learning of high-quality policies on goal-conditioned continuous control from pixel observations in only 1-2 million environment steps.

arXiv commentPublished at ICLR 2020

References in corpus (27)

• Adam: A Method for Stochastic Optimization
• Continuous control with deep reinforcement learning
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• The information bottleneck method
• Addressing Function Approximation Error in Actor-Critic Methods
• Soft Actor-Critic Algorithms and Applications
• Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• DeepMind Control Suite
• Deep Variational Information Bottleneck
• Distributed Distributional Deterministic Policy Gradients
• Learning Dexterous In-Hand Manipulation
• FeUdal Networks for Hierarchical Reinforcement Learning
• Stochastic Neural Networks for Hierarchical Reinforcement Learning
• Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishing
• DARLA: Improving Zero-Shot Transfer in Reinforcement Learning
• Deep Successor Reinforcement Learning
• Self-Consistent Trajectory Autoencoder: Hierarchical Reinforcement Learning with Trajectory Embeddings
• Model-Predictive Policy Learning with Uncertainty Regularization for Driving in Dense Traffic
• Latent Space Policies for Hierarchical Reinforcement Learning
• Near-Optimal Representation Learning for Hierarchical Reinforcement Learning
• PVEs: Position-Velocity Encoders for Unsupervised Learning of Structured State Representations
• Continual State Representation Learning for Reinforcement Learning using Generative Replay
• The Natural Language of Actions

Cited by in corpus (4)

• Learning Action-Transferable Policy with Action Embedding
• Jointly-Learned State-Action Embedding for Efficient Reinforcement Learning
• HyAR: Addressing Discrete-Continuous Action Reinforcement Learning via Hybrid Action Representation
• A Self-Supervised Auxiliary Loss for Deep RL in Partially Observable Settings