Model-Invariant State Abstractions for Model-Based Reinforcement Learning
arXiv:2102.09850
Abstract
Accuracy and generalization of dynamics models is key to the success of model-based reinforcement learning (MBRL). As the complexity of tasks increases, so does the sample inefficiency of learning accurate dynamics models. However, many complex tasks also exhibit sparsity in the dynamics, i.e., actions have only a local effect on the system dynamics. In this paper, we exploit this property with a causal invariance perspective in the single-task setting, introducing a new type of state abstraction called \textit{model-invariance}. Unlike previous forms of state abstractions, a model-invariance state abstraction leverages causal sparsity over state variables. This allows for compositional generalization to unseen states, something that non-factored forms of state abstractions cannot do. We prove that an optimal policy can be learned over this model-invariance state abstraction and show improved generalization in a simple toy domain. Next, we propose a practical method to approximately learn a model-invariant representation for complex domains and validate our approach by showing improved modelling performance over standard maximum likelihood approaches on challenging tasks, such as the MuJoCo-based Humanoid. Finally, within the MBRL setting we show strong performance gains with respect to sample efficiency across a host of other continuous control tasks.
References in corpus (16)
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Decision-Theoretic Planning: Structural Assumptions and Computational Leverage
- Invariant Risk Minimization
- Model-Ensemble Trust-Region Policy Optimization
- MOPO: Model-based Offline Policy Optimization
- MOReL : Model-Based Offline Reinforcement Learning
- The Risks of Invariant Risk Minimization
- Exploring Model-based Planning with Policy Networks
- Learning explanations that are hard to vary
- Model-based RL in Contextual Decision Processes: PAC bounds and Exponential Improvements over Model-free Approaches
- Provably efficient RL with Rich Observations via Latent State Decoding
- Information-Theoretic Considerations in Batch Reinforcement Learning
- Near-optimal Reinforcement Learning in Factored MDPs
- Representation Learning via Invariant Causal Mechanisms
- Invariant Causal Prediction for Block MDPs
- On the model-based stochastic value gradient for continuous reinforcement learning