Action-Sufficient State Representation Learning for Control with Structural Constraints
arXiv:2110.05721
Abstract
Perceived signals in real-world scenarios are usually high-dimensional and noisy, and finding and using their representation that contains essential and sufficient information required by downstream decision-making tasks will help improve computational efficiency and generalization ability in the tasks. In this paper, we focus on partially observable environments and propose to learn a minimal set of state representations that capture sufficient information for decision-making, termed \textit{Action-Sufficient state Representations} (ASRs). We build a generative environment model for the structural relationships among variables in the system and present a principled way to characterize ASRs based on structural constraints and the goal of maximizing cumulative reward in policy learning. We then develop a structured sequential Variational Auto-Encoder to estimate the environment model and extract ASRs. Our empirical results on CarRacing and VizDoom demonstrate a clear advantage of learning and using ASRs for policy learning. Moreover, the estimated environment model and ASRs allow learning behaviors from imagined outcomes in the compact latent space to improve sample efficiency.
References in corpus (18)
- Continuous control with deep reinforcement learning
- Playing Atari with Deep Reinforcement Learning
- The Kinetics Human Action Video Dataset
- The information bottleneck method
- Model-Based Reinforcement Learning for Atari
- State Representation Learning for Control: An Overview
- Deep Kalman Filters
- Dream to Control: Learning Behaviors by Latent Imagination
- SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning
- Causality for Machine Learning
- Self-Supervised Visual Planning with Temporal Skip Connections
- The Consciousness Prior
- Deep Successor Reinforcement Learning
- Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Data
- Independently Controllable Factors
- Causal Discovery and Forecasting in Nonstationary Environments with State-Space Models
- Temporal Difference Variational Auto-Encoder
- Predictive Coding for Locally-Linear Control