Spatial Broadcast Decoder: A Simple Architecture for Learning Disentangled Representations in VAEs

Abstract

We present a simple neural rendering architecture that helps variational autoencoders (VAEs) learn disentangled representations. Instead of the deconvolutional network typically used in the decoder of VAEs, we tile (broadcast) the latent vector across space, concatenate fixed X- and Y-"coordinate" channels, and apply a fully convolutional network with 1x1 stride. This provides an architectural prior for dissociating positional from non-positional features in the latent distribution of VAEs, yet without providing any explicit supervision to this effect. We show that this architecture, which we term the Spatial Broadcast decoder, improves disentangling, reconstruction accuracy, and generalization to held-out regions in data space. It provides a particularly dramatic benefit when applied to datasets with small objects. We also emphasize a method for visualizing learned latent spaces that helped us diagnose our models and may prove useful for others aiming to assess data representations. Finally, we show the Spatial Broadcast Decoder is complementary to state-of-the-art (SOTA) disentangling techniques and when incorporated improves their performance.

References in corpus (12)

• Auto-Encoding Variational Bayes
• Convolutional Sequence to Sequence Learning
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• DRAW: A Recurrent Neural Network For Image Generation
• Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations
• Deep Image Prior
• Disentangling by Factorising
• Deep Learning: A Critical Appraisal
• Towards a Definition of Disentangled Representations
• Understanding disentangling in $\beta$-VAE
• Learning Deep Disentangled Embeddings with the F-Statistic Loss
• SCAN: Learning Hierarchical Compositional Visual Concepts

Cited by in corpus (36)

• Multi-Object Representation Learning with Iterative Variational Inference
• Object-Centric Learning with Slot Attention
• MONet: Unsupervised Scene Decomposition and Representation
• GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations
• SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decomposition
• Reconstructing continuous distributions of 3D protein structure from cryo-EM images
• Localized convolutional neural networks for geospatial wind forecasting
• GENESIS-V2: Inferring Unordered Object Representations without Iterative Refinement
• Physics-as-Inverse-Graphics: Unsupervised Physical Parameter Estimation from Video
• SIMONe: View-Invariant, Temporally-Abstracted Object Representations via Unsupervised Video Decomposition
• Generative Neurosymbolic Machines
• Unsupervised Video Decomposition using Spatio-temporal Iterative Inference
• Reconstruction Bottlenecks in Object-Centric Generative Models
• Product Kanerva Machines: Factorized Bayesian Memory
• Learning Object-Centric Video Models by Contrasting Sets
• Generalization and Robustness Implications in Object-Centric Learning
• Guiding Video Prediction with Explicit Procedural Knowledge
• Triggering Dark Showers with Conditional Dual Auto-Encoders
• Benchmarking Unsupervised Object Representations for Video Sequences
• Generative Hierarchical Models for Parts, Objects, and Scenes
• Robust Feature Disentanglement in Imaging Data via Joint Invariant Variational Autoencoders: from Cards to Atoms
• Disentangled Relational Representations for Explaining and Learning from Demonstration
• Pre-training of Deep RL Agents for Improved Learning under Domain Randomization
• Visual Reasoning in Object-Centric Deep Neural Networks: A Comparative Cognition Approach
• SyMetric: Measuring the Quality of Learnt Hamiltonian Dynamics Inferred from Vision
• Disentangling ferroelectric wall dynamics and identification of pinning mechanisms via deep learning
• Learning from Demonstration with Weakly Supervised Disentanglement
• Reasoning-Modulated Representations
• Conditional Object-Centric Learning from Video
• Boxhead: A Dataset for Learning Hierarchical Representations
• A Proposal for Intelligent Agents with Episodic Memory
• Deep Variational Luenberger-type Observer for Stochastic Video Prediction
• Local Disentanglement in Variational Auto-Encoders Using Jacobian $L_1$ Regularization
• Group Equivariant Subsampling
• Learning to Give Checkable Answers with Prover-Verifier Games
• Illiterate DALL-E Learns to Compose