Deforming Autoencoders: Unsupervised Disentangling of Shape and Appearance

Abstract

In this work we introduce Deforming Autoencoders, a generative model for images that disentangles shape from appearance in an unsupervised manner. As in the deformable template paradigm, shape is represented as a deformation between a canonical coordinate system (`template') and an observed image, while appearance is modeled in `canonical', template, coordinates, thus discarding variability due to deformations. We introduce novel techniques that allow this approach to be deployed in the setting of autoencoders and show that this method can be used for unsupervised group-wise image alignment. We show experiments with expression morphing in humans, hands, and digits, face manipulation, such as shape and appearance interpolation, as well as unsupervised landmark localization. A more powerful form of unsupervised disentangling becomes possible in template coordinates, allowing us to successfully decompose face images into shading and albedo, and further manipulate face images.

17 pages including references, plus 12 pages appendix. Video available at : https://youtu.be/Oi7pyxKkF1g Code will be made available soon

References in corpus (2)

• InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets
• Learning Deep Representation for Face Alignment with Auxiliary Attributes

Cited by in corpus (25)

• Unsupervised Learning of Object Keypoints for Perception and Control
• Equivariant Transformer Networks
• Deformable Generator Networks: Unsupervised Disentanglement of Appearance and Geometry
• Exposing Deep-faked Videos by Anomalous Co-motion Pattern Detection
• Textured Neural Avatars
• Unsupervised Learning of Probably Symmetric Deformable 3D Objects from Images in the Wild
• Coordinate-based Texture Inpainting for Pose-Guided Image Generation
• High Fidelity Face Manipulation with Extreme Poses and Expressions
• Autoencoder Image Interpolation by Shaping the Latent Space
• Product of Orthogonal Spheres Parameterization for Disentangled Representation Learning
• FedDis: Disentangled Federated Learning for Unsupervised Brain Pathology Segmentation
• Make a Face: Towards Arbitrary High Fidelity Face Manipulation
• Deformation Flow Based Two-Stream Network for Lip Reading
• Unsupervised Learning of Landmarks based on Inter-Intra Subject Consistencies
• Latent feature disentanglement for 3D meshes
• Walking the Tightrope: An Investigation of the Convolutional Autoencoder Bottleneck
• Affine Variational Autoencoders: An Efficient Approach for Improving Generalization and Robustness to Distribution Shift
• Slim DensePose: Thrifty Learning from Sparse Annotations and Motion Cues
• Unsupervised Shape and Pose Disentanglement for 3D Meshes
• Learning to Aggregate and Personalize 3D Face from In-the-Wild Photo Collection
• Photo-Geometric Autoencoding to Learn 3D Objects from Unlabelled Images
• An Efficient Integration of Disentangled Attended Expression and Identity FeaturesFor Facial Expression Transfer andSynthesis
• Realistic Face Reenactment via Self-Supervised Disentangling of Identity and Pose
• On Equivariant and Invariant Learning of Object Landmark Representations
• Image-to-Video Generation via 3D Facial Dynamics