Attention-Aware Face Hallucination via Deep Reinforcement Learning

Abstract

Face hallucination is a domain-specific super-resolution problem with the goal to generate high-resolution (HR) faces from low-resolution (LR) input images. In contrast to existing methods that often learn a single patch-to-patch mapping from LR to HR images and are regardless of the contextual interdependency between patches, we propose a novel Attention-aware Face Hallucination (Attention-FH) framework which resorts to deep reinforcement learning for sequentially discovering attended patches and then performing the facial part enhancement by fully exploiting the global interdependency of the image. Specifically, in each time step, the recurrent policy network is proposed to dynamically specify a new attended region by incorporating what happened in the past. The state (i.e., face hallucination result for the whole image) can thus be exploited and updated by the local enhancement network on the selected region. The Attention-FH approach jointly learns the recurrent policy network and local enhancement network through maximizing the long-term reward that reflects the hallucination performance over the whole image. Therefore, our proposed Attention-FH is capable of adaptively personalizing an optimal searching path for each face image according to its own characteristic. Extensive experiments show our approach significantly surpasses the state-of-the-arts on in-the-wild faces with large pose and illumination variations.

References in corpus (7)

• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• DRAW: A Recurrent Neural Network For Image Generation
• DeepID3: Face Recognition with Very Deep Neural Networks
• Dynamic Memory Networks for Visual and Textual Question Answering
• Naive-Deep Face Recognition: Touching the Limit of LFW Benchmark or Not?
• Deep Cascaded Bi-Network for Face Hallucination
• Global-Local Face Upsampling Network

Cited by in corpus (11)

• Deep Reinforcement Learning: An Overview
• SiGAN: Siamese Generative Adversarial Network for Identity-Preserving Face Hallucination
• FSRNet: End-to-End Learning Face Super-Resolution with Facial Priors
• Crafting a Toolchain for Image Restoration by Deep Reinforcement Learning
• Pro-UIGAN: Progressive Face Hallucination from Occluded Thumbnails
• Non-locally Enhanced Encoder-Decoder Network for Single Image De-raining
• Deep Learning-based Face Super-Resolution: A Survey
• Outline Objects using Deep Reinforcement Learning
• CIRL: Controllable Imitative Reinforcement Learning for Vision-based Self-driving
• To learn image super-resolution, use a GAN to learn how to do image degradation first
• VidFace: A Full-Transformer Solver for Video FaceHallucination with Unaligned Tiny Snapshots