PlayVirtual: Augmenting Cycle-Consistent Virtual Trajectories for Reinforcement Learning
arXiv:2106.04152
Abstract
Learning good feature representations is important for deep reinforcement learning (RL). However, with limited experience, RL often suffers from data inefficiency for training. For un-experienced or less-experienced trajectories (i.e., state-action sequences), the lack of data limits the use of them for better feature learning. In this work, we propose a novel method, dubbed PlayVirtual, which augments cycle-consistent virtual trajectories to enhance the data efficiency for RL feature representation learning. Specifically, PlayVirtual predicts future states in the latent space based on the current state and action by a dynamics model and then predicts the previous states by a backward dynamics model, which forms a trajectory cycle. Based on this, we augment the actions to generate a large amount of virtual state-action trajectories. Being free of groudtruth state supervision, we enforce a trajectory to meet the cycle consistency constraint, which can significantly enhance the data efficiency. We validate the effectiveness of our designs on the Atari and DeepMind Control Suite benchmarks. Our method achieves the state-of-the-art performance on both benchmarks.
arXiv commentAccepted to NeurIPS 2021
References in corpus (21)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Soft Actor-Critic Algorithms and Applications
- Dueling Network Architectures for Deep Reinforcement Learning
- Learning deep representations by mutual information estimation and maximization
- Dota 2 with Large Scale Deep Reinforcement Learning
- Learning to Discover Cross-Domain Relations with Generative Adversarial Networks
- QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
- DeepMind Control Suite
- Rainbow: Combining Improvements in Deep Reinforcement Learning
- Learning Latent Dynamics for Planning from Pixels
- SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning
- Reinforcement Learning in Healthcare: A Survey
- When to use parametric models in reinforcement learning?
- Loss is its own Reward: Self-Supervision for Reinforcement Learning
- Decoupling Representation Learning from Reinforcement Learning
- Bootstrap Latent-Predictive Representations for Multitask Reinforcement Learning
- Predictive Information Accelerates Learning in RL
- Return-Based Contrastive Representation Learning for Reinforcement Learning
- Forward-Backward Reinforcement Learning
- Learning Cross-Domain Correspondence for Control with Dynamics Cycle-Consistency