Explicitly disentangling image content from translation and rotation with spatial-VAE

Abstract

Given an image dataset, we are often interested in finding data generative factors that encode semantic content independently from pose variables such as rotation and translation. However, current disentanglement approaches do not impose any specific structure on the learned latent representations. We propose a method for explicitly disentangling image rotation and translation from other unstructured latent factors in a variational autoencoder (VAE) framework. By formulating the generative model as a function of the spatial coordinate, we make the reconstruction error differentiable with respect to latent translation and rotation parameters. This formulation allows us to train a neural network to perform approximate inference on these latent variables while explicitly constraining them to only represent rotation and translation. We demonstrate that this framework, termed spatial-VAE, effectively learns latent representations that disentangle image rotation and translation from content and improves reconstruction over standard VAEs on several benchmark datasets, including applications to modeling continuous 2-D views of proteins from single particle electron microscopy and galaxies in astronomical images.

arXiv comment11 pages, 6 figures, to appear in the 33rd Conference on Neural Information Processing Systems (NeurIPS 2019)

Cited by in corpus (9)

• Exploring order parameters and dynamic processes in disordered systems via variational autoencoders
• Exploring particle dynamics during self-organization processes via rotationally invariant latent representations
• Toward Decoding the Relationship between Domain Structure and Functionality in Ferroelectrics via Hidden Latent Variables
• Automated and Autonomous Experiment in Electron and Scanning Probe Microscopy
• Improved Transformer for High-Resolution GANs
• Unsupervised Machine Learning Discovery of Chemical and Physical Transformation Pathways from Imaging Data
• Disentangling ferroelectric wall dynamics and identification of pinning mechanisms via deep learning
• Disentangling semantic features of macromolecules in Cryo-Electron Tomography
• Physical discovery in representation learning via conditioning on prior knowledge: applications for ferroelectric domain dynamics