SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning

Abstract

Model-based reinforcement learning (RL) has proven to be a data efficient approach for learning control tasks but is difficult to utilize in domains with complex observations such as images. In this paper, we present a method for learning representations that are suitable for iterative model-based policy improvement, even when the underlying dynamical system has complex dynamics and image observations, in that these representations are optimized for inferring simple dynamics and cost models given data from the current policy. This enables a model-based RL method based on the linear-quadratic regulator (LQR) to be used for systems with image observations. We evaluate our approach on a range of robotics tasks, including manipulation with a real-world robotic arm directly from images. We find that our method produces substantially better final performance than other model-based RL methods while being significantly more efficient than model-free RL.

ICML 2019. Project website: https://sites.google.com/view/icml19solar

Cited by in corpus (64)

• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Solving Rubik's Cube with a Robot Hand
• Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control
• Dream to Control: Learning Behaviors by Latent Imagination
• DeepMDP: Learning Continuous Latent Space Models for Representation Learning
• COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity-Driven Exploration
• Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model
• Planning to Explore via Self-Supervised World Models
• Online Service Migration in Mobile Edge with Incomplete System Information: A Deep Recurrent Actor-Critic Learning Approach
• Bootstrap Latent-Predictive Representations for Multitask Reinforcement Learning
• Reinforcement Learning for Robotic Manipulation using Simulated Locomotion Demonstrations
• Entity Abstraction in Visual Model-Based Reinforcement Learning
• FitVid: Overfitting in Pixel-Level Video Prediction
• Long-Horizon Visual Planning with Goal-Conditioned Hierarchical Predictors
• Context-aware Dynamics Model for Generalization in Model-Based Reinforcement Learning
• Action and Perception as Divergence Minimization
• Mastering Atari with Discrete World Models
• How to pick the domain randomization parameters for sim-to-real transfer of reinforcement learning policies?
• Objective Mismatch in Model-based Reinforcement Learning
• Landmark-Guided Subgoal Generation in Hierarchical Reinforcement Learning
• Integrating Contrastive Learning with Dynamic Models for Reinforcement Learning from Images
• Offline Reinforcement Learning from Images with Latent Space Models
• Contextual Imagined Goals for Self-Supervised Robotic Learning
• PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training
• Clockwork Variational Autoencoders
• ModelLight: Model-Based Meta-Reinforcement Learning for Traffic Signal Control
• DreamerPro: Reconstruction-Free Model-Based Reinforcement Learning with Prototypical Representations
• The Value Equivalence Principle for Model-Based Reinforcement Learning
• Floyd-Warshall Reinforcement Learning: Learning from Past Experiences to Reach New Goals
• PlayVirtual: Augmenting Cycle-Consistent Virtual Trajectories for Reinforcement Learning
• Control-Aware Representations for Model-based Reinforcement Learning
• Prediction, Consistency, Curvature: Representation Learning for Locally-Linear Control
• Learning Actionable Representations with Goal-Conditioned Policies
• Mutual Information Maximization for Robust Plannable Representations
• The Differentiable Cross-Entropy Method
• Greedy Hierarchical Variational Autoencoders for Large-Scale Video Prediction
• Acceleration of Actor-Critic Deep Reinforcement Learning for Visual Grasping in Clutter by State Representation Learning Based on Disentanglement of a Raw Input Image
• Imagined Value Gradients: Model-Based Policy Optimization with Transferable Latent Dynamics Models
• MUSBO: Model-based Uncertainty Regularized and Sample Efficient Batch Optimization for Deployment Constrained Reinforcement Learning
• Visual Adversarial Imitation Learning using Variational Models
• Weakly-Supervised Reinforcement Learning for Controllable Behavior
• Action-Sufficient State Representation Learning for Control with Structural Constraints
• Learning to Manipulate Object Collections Using Grounded State Representations
• Predictive Coding for Locally-Linear Control
• Value Function Spaces: Skill-Centric State Abstractions for Long-Horizon Reasoning
• Feedback from Pixels: Output Regulation via Learning-Based Scene View Synthesis
• Learning Latent State Spaces for Planning through Reward Prediction
• Deep Latent Competition: Learning to Race Using Visual Control Policies in Latent Space
• Reinforcement Learning with Efficient Active Feature Acquisition
• MELD: Meta-Reinforcement Learning from Images via Latent State Models
• Planning in Learned Latent Action Spaces for Generalizable Legged Locomotion
• Sample Efficient Reinforcement Learning with Partial Dynamics Knowledge
• LS3: Latent Space Safe Sets for Long-Horizon Visuomotor Control of Sparse Reward Iterative Tasks
• Assisted Perception: Optimizing Observations to Communicate State
• Towards Robust Bisimulation Metric Learning
• Learning to Plan Optimistically: Uncertainty-Guided Deep Exploration via Latent Model Ensembles
• VP-GO: a "light" action-conditioned visual prediction model
• Low-Variance Policy Gradient Estimation with World Models
• Planning from Images with Deep Latent Gaussian Process Dynamics
• Dynamic Horizon Value Estimation for Model-based Reinforcement Learning
• End-to-end grasping policies for human-in-the-loop robots via deep reinforcement learning
• Proper Value Equivalence
• Learn Proportional Derivative Controllable Latent Space from Pixels
• Deterministic Value-Policy Gradients