Generative Neurosymbolic Machines

Abstract

Reconciling symbolic and distributed representations is a crucial challenge that can potentially resolve the limitations of current deep learning. Remarkable advances in this direction have been achieved recently via generative object-centric representation models. While learning a recognition model that infers object-centric symbolic representations like bounding boxes from raw images in an unsupervised way, no such model can provide another important ability of a generative model, i.e., generating (sampling) according to the structure of learned world density. In this paper, we propose Generative Neurosymbolic Machines, a generative model that combines the benefits of distributed and symbolic representations to support both structured representations of symbolic components and density-based generation. These two crucial properties are achieved by a two-layer latent hierarchy with the global distributed latent for flexible density modeling and the structured symbolic latent map. To increase the model flexibility in this hierarchical structure, we also propose the StructDRAW prior. In experiments, we show that the proposed model significantly outperforms the previous structured representation models as well as the state-of-the-art non-structured generative models in terms of both structure accuracy and image generation quality. Our code, datasets, and trained models are available at https://github.com/JindongJiang/GNM

arXiv commentAccepted to NeurIPS as a spotlight paper

References in corpus (18)

• Auto-Encoding Variational Bayes
• Conditional Image Generation with PixelCNN Decoders
• Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations
• Disentangling by Factorising
• NVAE: A Deep Hierarchical Variational Autoencoder
• Learning Latent Dynamics for Planning from Pixels
• Multi-Object Representation Learning with Iterative Variational Inference
• MONet: Unsupervised Scene Decomposition and Representation
• GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations
• Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactions
• Continuously Differentiable Exponential Linear Units
• Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects
• SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decomposition
• Spatial Broadcast Decoder: A Simple Architecture for Learning Disentangled Representations in VAEs
• Metacontrol for Adaptive Imagination-Based Optimization
• Entity Abstraction in Visual Model-Based Reinforcement Learning
• SCALOR: Generative World Models with Scalable Object Representations
• A Perspective on Objects and Systematic Generalization in Model-Based RL