FSRNet: End-to-End Learning Face Super-Resolution with Facial Priors

Abstract

Face Super-Resolution (SR) is a domain-specific super-resolution problem. The specific facial prior knowledge could be leveraged for better super-resolving face images. We present a novel deep end-to-end trainable Face Super-Resolution Network (FSRNet), which makes full use of the geometry prior, i.e., facial landmark heatmaps and parsing maps, to super-resolve very low-resolution (LR) face images without well-aligned requirement. Specifically, we first construct a coarse SR network to recover a coarse high-resolution (HR) image. Then, the coarse HR image is sent to two branches: a fine SR encoder and a prior information estimation network, which extracts the image features, and estimates landmark heatmaps/parsing maps respectively. Both image features and prior information are sent to a fine SR decoder to recover the HR image. To further generate realistic faces, we propose the Face Super-Resolution Generative Adversarial Network (FSRGAN) to incorporate the adversarial loss into FSRNet. Moreover, we introduce two related tasks, face alignment and parsing, as the new evaluation metrics for face SR, which address the inconsistency of classic metrics w.r.t. visual perception. Extensive benchmark experiments show that FSRNet and FSRGAN significantly outperforms state of the arts for very LR face SR, both quantitatively and qualitatively. Code will be made available upon publication.

arXiv commentChen and Tai contributed equally to this paper

References in corpus (10)

• Stacked Hourglass Networks for Human Pose Estimation
• MemNet: A Persistent Memory Network for Image Restoration
• Generative Face Completion
• Adversarial PoseNet: A Structure-aware Convolutional Network for Human Pose Estimation
• Attention-Aware Face Hallucination via Deep Reinforcement Learning
• Scale-Aware Face Detection
• Pose-Invariant Face Alignment with a Single CNN
• Global-Local Face Upsampling Network
• Learning to Hallucinate Face Images via Component Generation and Enhancement
• Adversarial Learning of Structure-Aware Fully Convolutional Networks for Landmark Localization

Cited by in corpus (5)

• Face Recognition in Low Quality Images: A Survey
• CaricatureShop: Personalized and Photorealistic Caricature Sketching
• Reconstructing High-resolution Turbulent Flows Using Physics-Guided Neural Networks
• Implicit Subspace Prior Learning for Dual-Blind Face Restoration
• Joint Face Hallucination and Deblurring via Structure Generation and Detail Enhancement