Local Disentanglement in Variational Auto-Encoders Using Jacobian $L_1$ Regularization

Abstract

There have been many recent advances in representation learning; however, unsupervised representation learning can still struggle with model identification issues related to rotations of the latent space. Variational Auto-Encoders (VAEs) and their extensions such as $\beta$-VAEs have been shown to improve local alignment of latent variables with PCA directions, which can help to improve model disentanglement under some conditions. Borrowing inspiration from Independent Component Analysis (ICA) and sparse coding, we propose applying an $L_1$ loss to the VAE's generative Jacobian during training to encourage local latent variable alignment with independent factors of variation in images of multiple objects or images with multiple parts. We demonstrate our results on a variety of datasets, giving qualitative and quantitative results using information theoretic and modularity measures that show our added $L_1$ cost encourages local axis alignment of the latent representation with individual factors of variation.

arXiv comment17 pages, 10 figures, NeurIPS 2021 camera ready

References in corpus (8)

• Scikit-learn: Machine Learning in Python
• Auto-Encoding Variational Bayes
• Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations
• Spatial Broadcast Decoder: A Simple Architecture for Learning Disentangled Representations in VAEs
• On the Transfer of Inductive Bias from Simulation to the Real World: a New Disentanglement Dataset
• Robust Learning with Jacobian Regularization
• On Implicit Regularization in $\beta$-VAEs
• Improved Training of Sparse Coding Variational Autoencoder via Weight Normalization