Learning Representations for Pixel-based Control: What Matters and Why?

Abstract

Learning representations for pixel-based control has garnered significant attention recently in reinforcement learning. A wide range of methods have been proposed to enable efficient learning, leading to sample complexities similar to those in the full state setting. However, moving beyond carefully curated pixel data sets (centered crop, appropriate lighting, clear background, etc.) remains challenging. In this paper, we adopt a more difficult setting, incorporating background distractors, as a first step towards addressing this challenge. We present a simple baseline approach that can learn meaningful representations with no metric-based learning, no data augmentations, no world-model learning, and no contrastive learning. We then analyze when and why previously proposed methods are likely to fail or reduce to the same performance as the baseline in this harder setting and why we should think carefully about extending such methods beyond the well curated environments. Our results show that finer categorization of benchmarks on the basis of characteristics like density of reward, planning horizon of the problem, presence of task-irrelevant components, etc., is crucial in evaluating algorithms. Based on these observations, we propose different metrics to consider when evaluating an algorithm on benchmark tasks. We hope such a data-centric view can motivate researchers to rethink representation learning when investigating how to best apply RL to real-world tasks.

References in corpus (25)

• A Simple Framework for Contrastive Learning of Visual Representations
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Bootstrap your own latent: A new approach to self-supervised Learning
• The Kinetics Human Action Video Dataset
• Barlow Twins: Self-Supervised Learning via Redundancy Reduction
• DeepMind Control Suite
• Action-Conditional Video Prediction using Deep Networks in Atari Games
• Model-Based Reinforcement Learning for Atari
• Learning Latent Dynamics for Planning from Pixels
• Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images
• Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
• What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study
• DeepMDP: Learning Continuous Latent Space Models for Representation Learning
• Data-Efficient Reinforcement Learning with Self-Predictive Representations
• Reinforcement Learning with Prototypical Representations
• Invariant Causal Prediction for Block MDPs
• Natural Environment Benchmarks for Reinforcement Learning
• Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning
• Predictive Information Accelerates Learning in RL
• The Distracting Control Suite -- A Challenging Benchmark for Reinforcement Learning from Pixels
• Ecological Reinforcement Learning
• PAC Reinforcement Learning with Rich Observations
• Learning Discrete State Abstractions With Deep Variational Inference
• Learning Robust State Abstractions for Hidden-Parameter Block MDPs
• Model-Invariant State Abstractions for Model-Based Reinforcement Learning