Towards Nonlinear Disentanglement in Natural Data with Temporal Sparse Coding

Abstract

We construct an unsupervised learning model that achieves nonlinear disentanglement of underlying factors of variation in naturalistic videos. Previous work suggests that representations can be disentangled if all but a few factors in the environment stay constant at any point in time. As a result, algorithms proposed for this problem have only been tested on carefully constructed datasets with this exact property, leaving it unclear whether they will transfer to natural scenes. Here we provide evidence that objects in segmented natural movies undergo transitions that are typically small in magnitude with occasional large jumps, which is characteristic of a temporally sparse distribution. We leverage this finding and present SlowVAE, a model for unsupervised representation learning that uses a sparse prior on temporally adjacent observations to disentangle generative factors without any assumptions on the number of changing factors. We provide a proof of identifiability and show that the model reliably learns disentangled representations on several established benchmark datasets, often surpassing the current state-of-the-art. We additionally demonstrate transferability towards video datasets with natural dynamics, Natural Sprites and KITTI Masks, which we contribute as benchmarks for guiding disentanglement research towards more natural data domains.

arXiv commentICLR 2021. Code is available at https://github.com/bethgelab/slow_disentanglement. The first three authors, as well as the last two authors, contributed equally

References in corpus (36)

• Scikit-learn: Machine Learning in Python
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Representation Learning with Contrastive Predictive Coding
• MOT16: A Benchmark for Multi-Object Tracking
• NICE: Non-linear Independent Components Estimation
• Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations
• Deep Convolutional Inverse Graphics Network
• Density estimation using Real NVP
• Disentangling by Factorising
• YouTube-VOS: A Large-Scale Video Object Segmentation Benchmark
• Towards a Definition of Disentangled Representations
• Multi-Object Representation Learning with Iterative Variational Inference
• MONet: Unsupervised Scene Decomposition and Representation
• Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear ICA
• Discovering Hidden Factors of Variation in Deep Networks
• Flexibly Fair Representation Learning by Disentanglement
• Lagging Inference Networks and Posterior Collapse in Variational Autoencoders
• Learning Deep Disentangled Embeddings with the F-Statistic Loss
• BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling
• Weakly-Supervised Disentanglement Without Compromises
• Weakly Supervised Disentanglement with Guarantees
• On the Transfer of Inductive Bias from Simulation to the Real World: a New Disentanglement Dataset
• Disentanglement by Nonlinear ICA with General Incompressible-flow Networks (GIN)
• Nonlinear ICA Using Auxiliary Variables and Generalized Contrastive Learning
• On Disentangled Representations Learned From Correlated Data
• A Survey of Inductive Biases for Factorial Representation-Learning
• Unsupervised Model Selection for Variational Disentangled Representation Learning
• Challenges in Disentangling Independent Factors of Variation
• Entity Abstraction in Visual Model-Based Reinforcement Learning
• ICE-BeeM: Identifiable Conditional Energy-Based Deep Models Based on Nonlinear ICA
• Disentangling Space and Time in Video with Hierarchical Variational Auto-encoders
• CausalVAE: Structured Causal Disentanglement in Variational Autoencoder
• An Unsupervised Algorithm For Learning Lie Group Transformations
• Video Instance Segmentation
• Addressing the Topological Defects of Disentanglement via Distributed Operators
• Relative gradient optimization of the Jacobian term in unsupervised deep learning

Cited by in corpus (5)

• Learnable latent embeddings for joint behavioral and neural analysis
• On the Transfer of Disentangled Representations in Realistic Settings
• Learning Temporally Causal Latent Processes from General Temporal Data
• Visual Representation Learning Does Not Generalize Strongly Within the Same Domain
• Properties from Mechanisms: An Equivariance Perspective on Identifiable Representation Learning