Structured Generative Models for Scene Understanding
arXiv:2302.03531 · doi:10.1007/s11263-024-02316-z
Abstract
This position paper argues for the use of \emph{structured generative models} (SGMs) for the understanding of static scenes. This requires the reconstruction of a 3D scene from an input image (or a set of multi-view images), whereby the contents of the image(s) are causally explained in terms of models of instantiated objects, each with their own type, shape, appearance and pose, along with global variables like scene lighting and camera parameters. This approach also requires scene models which account for the co-occurrences and inter-relationships of objects in a scene. The SGM approach has the merits that it is compositional and generative, which lead to interpretability and editability. \\\\ To pursue the SGM agenda, we need models for objects and scenes, and approaches to carry out inference. We first review models for objects, which include ``things'' (object categories that have a well defined shape), and ``stuff'' (categories which have amorphous spatial extent). We then move on to review \emph{scene models} which describe the inter-relationships of objects. Perhaps the most challenging problem for SGMs is \emph{inference} of the objects, lighting and camera parameters, and scene inter-relationships from input consisting of a single or multiple images. We conclude with a discussion of issues that need addressing to advance the SGM agenda.
32 pages, 10 figures
References in corpus (16)
- Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
- Stochastic Backpropagation and Approximate Inference in Deep Generative Models
- GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models
- Gemini: A Family of Highly Capable Multimodal Models
- Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations
- NeRFactor: Neural Factorization of Shape and Reflectance Under an Unknown Illumination
- Attend, Infer, Repeat: Fast Scene Understanding with Generative Models
- Multi-Object Representation Learning with Iterative Variational Inference
- Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks
- GRAINS: Generative Recursive Autoencoders for INdoor Scenes
- ATISS: Autoregressive Transformers for Indoor Scene Synthesis
- Learning Object-Centric Representations of Multi-Object Scenes from Multiple Views
- Reduce, Reuse, Recycle: Compositional Generation with Energy-Based Diffusion Models and MCMC
- NeMo: Neural Mesh Models of Contrastive Features for Robust 3D Pose Estimation
- Learning Direct Optimization for Scene Understanding
- Inference and Learning for Generative Capsule Models