Supervision-by-Registration: An Unsupervised Approach to Improve the Precision of Facial Landmark Detectors

Abstract

In this paper, we present supervision-by-registration, an unsupervised approach to improve the precision of facial landmark detectors on both images and video. Our key observation is that the detections of the same landmark in adjacent frames should be coherent with registration, i.e., optical flow. Interestingly, the coherency of optical flow is a source of supervision that does not require manual labeling, and can be leveraged during detector training. For example, we can enforce in the training loss function that a detected landmark at frame${}_{t-1}$ followed by optical flow tracking from frame${}_{t-1}$ to frame${}_t$ should coincide with the location of the detection at frame${}_t$. Essentially, supervision-by-registration augments the training loss function with a registration loss, thus training the detector to have output that is not only close to the annotations in labeled images, but also consistent with registration on large amounts of unlabeled videos. End-to-end training with the registration loss is made possible by a differentiable Lucas-Kanade operation, which computes optical flow registration in the forward pass, and back-propagates gradients that encourage temporal coherency in the detector. The output of our method is a more precise image-based facial landmark detector, which can be applied to single images or video. With supervision-by-registration, we demonstrate (1) improvements in facial landmark detection on both images (300W, ALFW) and video (300VW, Youtube-Celebrities), and (2) significant reduction of jittering in video detections.

arXiv commentMinor modifications to the CVPR 2018 version (add missing references)

References in corpus (6)

• Deep Residual Learning for Image Recognition
• Stacked Hourglass Networks for Human Pose Estimation
• Convolutional Pose Machines
• Style Aggregated Network for Facial Landmark Detection
• More is Less: A More Complicated Network with Less Inference Complexity
• A Recurrent Encoder-Decoder Network for Sequential Face Alignment

Cited by in corpus (9)

• Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks
• Style Aggregated Network for Facial Landmark Detection
• Adaptive Wing Loss for Robust Face Alignment via Heatmap Regression
• 3D Multi-Object Tracking: A Baseline and New Evaluation Metrics
• Monocular 3D Object Detection with Pseudo-LiDAR Point Cloud
• Teacher Supervises Students How to Learn From Partially Labeled Images for Facial Landmark Detection
• Beyond Trade-off: Accelerate FCN-based Face Detector with Higher Accuracy
• 2D Wasserstein Loss for Robust Facial Landmark Detection
• GroundNet: Monocular Ground Plane Normal Estimation with Geometric Consistency