State Representation Learning for Control: An Overview

Abstract

Representation learning algorithms are designed to learn abstract features that characterize data. State representation learning (SRL) focuses on a particular kind of representation learning where learned features are in low dimension, evolve through time, and are influenced by actions of an agent. The representation is learned to capture the variation in the environment generated by the agent's actions; this kind of representation is particularly suitable for robotics and control scenarios. In particular, the low dimension characteristic of the representation helps to overcome the curse of dimensionality, provides easier interpretation and utilization by humans and can help improve performance and speed in policy learning algorithms such as reinforcement learning. This survey aims at covering the state-of-the-art on state representation learning in the most recent years. It reviews different SRL methods that involve interaction with the environment, their implementations and their applications in robotics control tasks (simulated or real). In particular, it highlights how generic learning objectives are differently exploited in the reviewed algorithms. Finally, it discusses evaluation methods to assess the representation learned and summarizes current and future lines of research.

References in corpus (31)

• Auto-Encoding Variational Bayes
• Continuous control with deep reinforcement learning
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• European Union regulations on algorithmic decision-making and a "right to explanation"
• Conditional Image Synthesis With Auxiliary Classifier GANs
• InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets
• Progressive Neural Networks
• The Arcade Learning Environment: An Evaluation Platform for General Agents
• Adversarial Feature Learning
• Adversarially Learned Inference
• DeepMind Control Suite
• Unsupervised Learning for Physical Interaction through Video Prediction
• Interactive Perception: Leveraging Action in Perception and Perception in Action
• Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images
• Learning to Poke by Poking: Experiential Learning of Intuitive Physics
• DARLA: Improving Zero-Shot Transfer in Reinforcement Learning
• Improving Exploration in Evolution Strategies for Deep Reinforcement Learning via a Population of Novelty-Seeking Agents
• Loss is its own Reward: Self-Supervision for Reinforcement Learning
• Strategic Attentive Writer for Learning Macro-Actions
• Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Data
• Unsupervised Learning of Goal Spaces for Intrinsically Motivated Goal Exploration
• Independently Controllable Factors
• DeepMind Lab
• PVEs: Position-Velocity Encoders for Unsupervised Learning of Structured State Representations
• Decoupling Dynamics and Reward for Transfer Learning
• Data-Efficient Learning of Feedback Policies from Image Pixels using Deep Dynamical Models
• Unsupervised state representation learning with robotic priors: a robustness benchmark
• Crossing the Reality Gap: a Short Introduction to the Transferability Approach
• Using PCA to Efficiently Represent State Spaces
• Deep Multimodal Representation Learning from Temporal Data
• Incremental Slow Feature Analysis: Adaptive and Episodic Learning from High-Dimensional Input Streams

Cited by in corpus (68)

• A survey on intrinsic motivation in reinforcement learning
• Recovering missing CFD data for high-order discretizations using deep neural networks and dynamics learning
• Data-Efficient Reinforcement Learning with Self-Predictive Representations
• Optimization of Pressure Management Strategies for Geological CO2 Sequestration Using Surrogate Model-based Reinforcement Learning
• DisCoRL: Continual Reinforcement Learning via Policy Distillation
• Testing of Deep Reinforcement Learning Agents with Surrogate Models
• Pretraining Representations for Data-Efficient Reinforcement Learning
• RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environments
• Defining Benchmarks for Continual Few-Shot Learning
• Unsupervised State Representation Learning in Atari
• Operationally meaningful representations of physical systems in neural networks
• An Empirical Analysis of Deep Learning for Cardinality Estimation
• Meta-Adaptive Nonlinear Control: Theory and Algorithms
• Graph-Structured Visual Imitation
• Continual State Representation Learning for Reinforcement Learning using Generative Replay
• Aligning Robot and Human Representations
• Deep Kernel Learning of Dynamical Models from High-Dimensional Noisy Data
• Model-Based Reinforcement Learning for Control of Strongly-Disturbed Unsteady Aerodynamic Flows
• Hierarchically Organized Latent Modules for Exploratory Search in Morphogenetic Systems
• Multimodal Information Bottleneck for Deep Reinforcement Learning with Multiple Sensors
• Adaptive Path-Integral Autoencoder: Representation Learning and Planning for Dynamical Systems
• Bayesian Optimization in Variational Latent Spaces with Dynamic Compression
• Training Larger Networks for Deep Reinforcement Learning
• Combining Federated Learning and Control: A Survey
• Geometric Entropic Exploration
• Which Mutual-Information Representation Learning Objectives are Sufficient for Control?
• Exploratory State Representation Learning
• Sparse Representation Learning with Modified q-VAE towards Minimal Realization of World Model
• Provable Representation Learning for Imitation with Contrastive Fourier Features
• Learning Actionable Representations with Goal-Conditioned Policies
• An A* Curriculum Approach to Reinforcement Learning for RGBD Indoor Robot Navigation
• Action-Sufficient State Representation Learning for Control with Structural Constraints
• Hierarchical principles of embodied reinforcement learning: A review
• On learning latent dynamics of the AUG plasma state
• Learning-based Adaptive Control of Quadruped Robots for Active Stabilization on Moving Platforms
• Should artificial agents ask for help in human-robot collaborative problem-solving?
• Explainable Autonomous Robots: A Survey and Perspective
• Low Dimensional State Representation Learning with Reward-shaped Priors
• Direct and inverse modeling of soft robots by learning a condensed FEM model
• Learning State Abstractions for Transfer in Continuous Control
• Pixyz: a Python library for developing deep generative models
• An Overview of Natural Language State Representation for Reinforcement Learning
• Learning Representations in Reinforcement Learning:An Information Bottleneck Approach
• Continual Learning: Tackling Catastrophic Forgetting in Deep Neural Networks with Replay Processes
• Domain-Adversarial and Conditional State Space Model for Imitation Learning
• Efficient Exploration in Constrained Environments with Goal-Oriented Reference Path
• Learning State Representations in Complex Systems with Multimodal Data
• Information State Embedding in Partially Observable Cooperative Multi-Agent Reinforcement Learning
• Evaluation of state representation methods in robot hand-eye coordination learning from demonstration
• From semantics to execution: Integrating action planning with reinforcement learning for robotic causal problem-solving
• Analytic Manifold Learning: Unifying and Evaluating Representations for Continuous Control
• Cloth Manipulation Planning on Basis of Mesh Representations with Incomplete Domain Knowledge and Voxel-to-Mesh Estimation
• Continual Learning in Deep Networks: an Analysis of the Last Layer
• Iterative Model-Based Reinforcement Learning Using Simulations in the Differentiable Neural Computer
• Autonomous Learning of Features for Control: Experiments with Embodied and Situated Agents
• Improving Reinforcement Learning Efficiency with Auxiliary Tasks in Non-Visual Environments: A Comparison
• Rogue-Gym: A New Challenge for Generalization in Reinforcement Learning
• Visual Radial Basis Q-Network
• State Representation Learning from Demonstration
• Capacity-Aware Planning and Scheduling in Budget-Constrained Multi-Agent MDPs: A Meta-RL Approach
• Supervise Thyself: Examining Self-Supervised Representations in Interactive Environments
• Learning in Sparse Rewards settings through Quality-Diversity algorithms
• The Greatest Teacher, Failure is: Using Reinforcement Learning for SFC Placement Based on Availability and Energy Consumption
• A representational framework for learning and encoding structurally enriched trajectories in complex agent environments
• RECON: Reducing Causal Confusion with Human-Placed Markers
• Learning Representations in Video Game Agents with Supervised Contrastive Imitation Learning
• Estimating Disentangled Belief about Hidden State and Hidden Task for Meta-RL
• Planning from Images with Deep Latent Gaussian Process Dynamics