Entity Abstraction in Visual Model-Based Reinforcement Learning

Abstract

This paper tests the hypothesis that modeling a scene in terms of entities and their local interactions, as opposed to modeling the scene globally, provides a significant benefit in generalizing to physical tasks in a combinatorial space the learner has not encountered before. We present object-centric perception, prediction, and planning (OP3), which to the best of our knowledge is the first fully probabilistic entity-centric dynamic latent variable framework for model-based reinforcement learning that acquires entity representations from raw visual observations without supervision and uses them to predict and plan. OP3 enforces entity-abstraction -- symmetric processing of each entity representation with the same locally-scoped function -- which enables it to scale to model different numbers and configurations of objects from those in training. Our approach to solving the key technical challenge of grounding these entity representations to actual objects in the environment is to frame this variable binding problem as an inference problem, and we develop an interactive inference algorithm that uses temporal continuity and interactive feedback to bind information about object properties to the entity variables. On block-stacking tasks, OP3 generalizes to novel block configurations and more objects than observed during training, outperforming an oracle model that assumes access to object supervision and achieving two to three times better accuracy than a state-of-the-art video prediction model that does not exhibit entity abstraction.

arXiv commentAccepted at CoRL 2019

References in corpus (32)

• Auto-Encoding Variational Bayes
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Tutorial on Variational Autoencoders
• Deep Convolutional Inverse Graphics Network
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• Interaction Networks for Learning about Objects, Relations and Physics
• Learning Latent Dynamics for Planning from Pixels
• Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control
• Multi-Object Representation Learning with Iterative Variational Inference
• Stochastic Adversarial Video Prediction
• MONet: Unsupervised Scene Decomposition and Representation
• A Compositional Object-Based Approach to Learning Physical Dynamics
• Control of Memory, Active Perception, and Action in Minecraft
• Schema Networks: Zero-shot Transfer with a Generative Causal Model of Intuitive Physics
• SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning
• Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactions
• Unsupervised Learning of Object Keypoints for Perception and Control
• Hierarchical Long-term Video Prediction without Supervision
• Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects
• COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity-Driven Exploration
• Deep Set Prediction Networks
• Unsupervised Video Object Segmentation for Deep Reinforcement Learning
• Variational Inverse Control with Events: A General Framework for Data-Driven Reward Definition
• Reasoning About Physical Interactions with Object-Oriented Prediction and Planning
• Metacontrol for Adaptive Imagination-Based Optimization
• Robustness via Retrying: Closed-Loop Robotic Manipulation with Self-Supervised Learning
• Understanding Visual Concepts with Continuation Learning
• Iterative Amortized Inference
• Binding via Reconstruction Clustering
• A General Method for Amortizing Variational Filtering
• Task-Agnostic Dynamics Priors for Deep Reinforcement Learning
• Object-Oriented Dynamics Learning through Multi-Level Abstraction

Cited by in corpus (12)

• The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence
• Towards Nonlinear Disentanglement in Natural Data with Temporal Sparse Coding
• Structure in Deep Reinforcement Learning: A Survey and Open Problems
• Generative Neurosymbolic Machines
• SIMONe: View-Invariant, Temporally-Abstracted Object Representations via Unsupervised Video Decomposition
• AlignNet: Unsupervised Entity Alignment
• MUSBO: Model-based Uncertainty Regularized and Sample Efficient Batch Optimization for Deployment Constrained Reinforcement Learning
• Efficient State Abstraction using Object-centered Predicates for Manipulation Planning
• Better Set Representations For Relational Reasoning
• Self-supervised Visual Reinforcement Learning with Object-centric Representations
• Learning Transferable Concepts in Deep Reinforcement Learning
• Relevance-Guided Modeling of Object Dynamics for Reinforcement Learning