Self-supervised learning of a facial attribute embedding from video

Abstract

We propose a self-supervised framework for learning facial attributes by simply watching videos of a human face speaking, laughing, and moving over time. To perform this task, we introduce a network, Facial Attributes-Net (FAb-Net), that is trained to embed multiple frames from the same video face-track into a common low-dimensional space. With this approach, we make three contributions: first, we show that the network can leverage information from multiple source frames by predicting confidence/attention masks for each frame; second, we demonstrate that using a curriculum learning regime improves the learned embedding; finally, we demonstrate that the network learns a meaningful face embedding that encodes information about head pose, facial landmarks and facial expression, i.e. facial attributes, without having been supervised with any labelled data. We are comparable or superior to state-of-the-art self-supervised methods on these tasks and approach the performance of supervised methods.

arXiv commentTo appear in BMVC 2018. Supplementary material can be found at http://www.robots.ox.ac.uk/~vgg/research/unsup_learn_watch_faces/fabnet.html

References in corpus (16)

• VoxCeleb2: Deep Speaker Recognition
• AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild
• Deep Convolutional Inverse Graphics Network
• Learning Deep Representation for Face Alignment with Auxiliary Attributes
• Unsupervised Visual Representation Learning by Context Prediction
• Spatio-temporal video autoencoder with differentiable memory
• Unsupervised Learning of Disentangled Representations from Video
• Unsupervised Learning of Visual Representations using Videos
• Online Batch Selection for Faster Training of Neural Networks
• Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks
• Learning to See by Moving
• Deep Learning Face Attributes in the Wild
• Generic 3D Representation via Pose Estimation and Matching
• 3D Morphable Models as Spatial Transformer Networks
• You said that?
• Unsupervised learning of object frames by dense equivariant image labelling

Cited by in corpus (27)

• Unsupervised Learning of Object Keypoints for Perception and Control
• Multi-task head pose estimation in-the-wild
• Memory-augmented Dense Predictive Coding for Video Representation Learning
• SVFAP: Self-supervised Video Facial Affect Perceiver
• Families In Wild Multimedia: A Multimodal Database for Recognizing Kinship
• Learning Temporal Pose Estimation from Sparsely-Labeled Videos
• Contrastive Learning of Person-independent Representations for Facial Action Unit Detection
• Emotion Recognition in Speech using Cross-Modal Transfer in the Wild
• Bounded Residual Gradient Networks (BReG-Net) for Facial Affect Computing
• Unconstrained Facial Action Unit Detection via Latent Feature Domain
• SelfORE: Self-supervised Relational Feature Learning for Open Relation Extraction
• A Facial Affect Analysis System for Autism Spectrum Disorder
• BRULÈ: Barycenter-Regularized Unsupervised Landmark Extraction
• Make a Face: Towards Arbitrary High Fidelity Face Manipulation
• A Survey of Deep Facial Attribute Analysis
• Deformation Flow Based Two-Stream Network for Lip Reading
• Unsupervised Learning of Landmarks based on Inter-Intra Subject Consistencies
• Template NeRF: Towards Modeling Dense Shape Correspondences from Category-Specific Object Images
• Unsupervised Representation Learning for Gaze Estimation
• KeypointDeformer: Unsupervised 3D Keypoint Discovery for Shape Control
• Landmarks Augmentation with Manifold-Barycentric Oversampling
• Automatic Face Understanding: Recognizing Families in Photos
• On Equivariant and Invariant Learning of Object Landmark Representations
• Realistic Face Reenactment via Self-Supervised Disentangling of Identity and Pose
• Meta Auxiliary Learning for Facial Action Unit Detection
• Point-to-Point Video Generation
• Harnessing Geometric Constraints from Emotion Labels to improve Face Verification