Reinforcement Learning with Latent Flow

Abstract

Temporal information is essential to learning effective policies with Reinforcement Learning (RL). However, current state-of-the-art RL algorithms either assume that such information is given as part of the state space or, when learning from pixels, use the simple heuristic of frame-stacking to implicitly capture temporal information present in the image observations. This heuristic is in contrast to the current paradigm in video classification architectures, which utilize explicit encodings of temporal information through methods such as optical flow and two-stream architectures to achieve state-of-the-art performance. Inspired by leading video classification architectures, we introduce the Flow of Latents for Reinforcement Learning (Flare), a network architecture for RL that explicitly encodes temporal information through latent vector differences. We show that Flare (i) recovers optimal performance in state-based RL without explicit access to the state velocity, solely with positional state information, (ii) achieves state-of-the-art performance on pixel-based challenging continuous control tasks within the DeepMind control benchmark suite, namely quadruped walk, hopper hop, finger turn hard, pendulum swing, and walker run, and is the most sample efficient model-free pixel-based RL algorithm, outperforming the prior model-free state-of-the-art by 1.9X and 1.5X on the 500k and 1M step benchmarks, respectively, and (iv), when augmented over rainbow DQN, outperforms this state-of-the-art level baseline on 5 of 8 challenging Atari games at 100M time step benchmark.

References in corpus (18)

• Two-Stream Convolutional Networks for Action Recognition in Videos
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Prioritized Experience Replay
• Dueling Network Architectures for Deep Reinforcement Learning
• Dota 2 with Large Scale Deep Reinforcement Learning
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Noisy Networks for Exploration
• DeepMind Control Suite
• Learning Latent Dynamics for Planning from Pixels
• CARLA: An Open Urban Driving Simulator
• Reinforcement Learning with Augmented Data
• A Distributional Perspective on Reinforcement Learning
• Agent57: Outperforming the Atari Human Benchmark
• Dream to Control: Learning Behaviors by Latent Imagination
• Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
• Decoupling Representation Learning from Reinforcement Learning
• Data-Efficient Reinforcement Learning with Self-Predictive Representations