Unsupervised Learning of Object Landmarks through Conditional Image Generation

Abstract

We propose a method for learning landmark detectors for visual objects (such as the eyes and the nose in a face) without any manual supervision. We cast this as the problem of generating images that combine the appearance of the object as seen in a first example image with the geometry of the object as seen in a second example image, where the two examples differ by a viewpoint change and/or an object deformation. In order to factorize appearance and geometry, we introduce a tight bottleneck in the geometry-extraction process that selects and distils geometry-related features. Compared to standard image generation problems, which often use generative adversarial networks, our generation task is conditioned on both appearance and geometry and thus is significantly less ambiguous, to the point that adopting a simple perceptual loss formulation is sufficient. We demonstrate that our approach can learn object landmarks from synthetic image deformations or videos, all without manual supervision, while outperforming state-of-the-art unsupervised landmark detectors. We further show that our method is applicable to a large variety of datasets - faces, people, 3D objects, and digits - without any modifications.

arXiv commentIn NeurIPS 2018. Project page: http://www.robots.ox.ac.uk/~vgg/research/unsupervised_landmarks/

Cited by in corpus (40)

• Unsupervised Learning of Object Keypoints for Perception and Control
• Memory-augmented Dense Predictive Coding for Video Representation Learning
• KOVIS: Keypoint-based Visual Servoing with Zero-Shot Sim-to-Real Transfer for Robotics Manipulation
• On the Binding Problem in Artificial Neural Networks
• Pose Estimation for Robot Manipulators via Keypoint Optimization and Sim-to-Real Transfer
• Causal Discovery in Physical Systems from Videos
• An Explicit Local and Global Representation Disentanglement Framework with Applications in Deep Clustering and Unsupervised Object Detection
• S3K: Self-Supervised Semantic Keypoints for Robotic Manipulation via Multi-View Consistency
• Test-Time Personalization with a Transformer for Human Pose Estimation
• Unsupervised Landmark Learning from Unpaired Data
• Improving Generative Imagination in Object-Centric World Models
• Adversarial Partial Multi-Label Learning
• Cerberus: A Multi-headed Derenderer
• BRULÈ: Barycenter-Regularized Unsupervised Landmark Extraction
• Video Interpolation and Prediction with Unsupervised Landmarks
• Slot Contrastive Networks: A Contrastive Approach for Representing Objects
• Generative Hierarchical Features from Synthesizing Images
• Concept Learners for Few-Shot Learning
• Semi-supervised Keypoint Localization
• Unsupervised Learning of Landmarks based on Inter-Intra Subject Consistencies
• JOKR: Joint Keypoint Representation for Unsupervised Cross-Domain Motion Retargeting
• Unsupervised Visual Attention and Invariance for Reinforcement Learning
• Vision-based system identification and 3D keypoint discovery using dynamics constraints
• KeypointDeformer: Unsupervised 3D Keypoint Discovery for Shape Control
• Unsupervised Object Keypoint Learning using Local Spatial Predictability
• GaussiGAN: Controllable Image Synthesis with 3D Gaussians from Unposed Silhouettes
• Kinematic-Structure-Preserved Representation for Unsupervised 3D Human Pose Estimation
• Unsupervised Landmark Detection Based Spatiotemporal Motion Estimation for 4D Dynamic Medical Images
• GANSeg: Learning to Segment by Unsupervised Hierarchical Image Generation
• Self-Supervised Object Detection via Generative Image Synthesis
• Weakly Supervised Keypoint Discovery
• Human 3D keypoints via spatial uncertainty modeling
• Learning the Imaging Landmarks: Unsupervised Key point Detection in Lung Ultrasound Videos
• Unsupervised Part Segmentation through Disentangling Appearance and Shape
• Pose Recognition in the Wild: Animal pose estimation using Agglomerative Clustering and Contrastive Learning
• Learning of feature points without additional supervision improves reinforcement learning from images
• TransferI2I: Transfer Learning for Image-to-Image Translation from Small Datasets
• Cross-Identity Motion Transfer for Arbitrary Objects through Pose-Attentive Video Reassembling
• On Equivariant and Invariant Learning of Object Landmark Representations
• Video Autoencoder: self-supervised disentanglement of static 3D structure and motion