Improving Generative Imagination in Object-Centric World Models

Abstract

The remarkable recent advances in object-centric generative world models raise a few questions. First, while many of the recent achievements are indispensable for making a general and versatile world model, it is quite unclear how these ingredients can be integrated into a unified framework. Second, despite using generative objectives, abilities for object detection and tracking are mainly investigated, leaving the crucial ability of temporal imagination largely under question. Third, a few key abilities for more faithful temporal imagination such as multimodal uncertainty and situation-awareness are missing. In this paper, we introduce Generative Structured World Models (G-SWM). The G-SWM achieves the versatile world modeling not only by unifying the key properties of previous models in a principled framework but also by achieving two crucial new abilities, multimodal uncertainty and situation-awareness. Our thorough investigation on the temporal generation ability in comparison to the previous models demonstrates that G-SWM achieves the versatility with the best or comparable performance for all experiment settings including a few complex settings that have not been tested before.

Accepted in ICML 2020

References in corpus (16)

• Adam: A Method for Stochastic Optimization
• Auto-Encoding Variational Bayes
• Neural Message Passing for Quantum Chemistry
• On the difficulty of training Recurrent Neural Networks
• Categorical Reparameterization with Gumbel-Softmax
• MOT16: A Benchmark for Multi-Object Tracking
• Recurrent World Models Facilitate Policy Evolution
• Rainbow: Combining Improvements in Deep Reinforcement Learning
• Graph networks as learnable physics engines for inference and control
• Multi-Object Representation Learning with Iterative Variational Inference
• MONet: Unsupervised Scene Decomposition and Representation
• Unsupervised Learning of Object Keypoints for Perception and Control
• Continuously Differentiable Exponential Linear Units
• Unsupervised Learning of Object Landmarks through Conditional Image Generation
• Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects
• Structured Object-Aware Physics Prediction for Video Modeling and Planning