Synthesis of High-Quality Visible Faces from Polarimetric Thermal Faces using Generative Adversarial Networks

Abstract

The large domain discrepancy between faces captured in polarimetric (or conventional) thermal and visible domain makes cross-domain face verification a highly challenging problem for human examiners as well as computer vision algorithms. Previous approaches utilize either a two-step procedure (visible feature estimation and visible image reconstruction) or an input-level fusion technique, where different Stokes images are concatenated and used as a multi-channel input to synthesize the visible image given the corresponding polarimetric signatures. Although these methods have yielded improvements, we argue that input-level fusion alone may not be sufficient to realize the full potential of the available Stokes images. We propose a Generative Adversarial Networks (GAN) based multi-stream feature-level fusion technique to synthesize high-quality visible images from prolarimetric thermal images. The proposed network consists of a generator sub-network, constructed using an encoder-decoder network based on dense residual blocks, and a multi-scale discriminator sub-network. The generator network is trained by optimizing an adversarial loss in addition to a perceptual loss and an identity preserving loss to enable photo realistic generation of visible images while preserving discriminative characteristics. An extended dataset consisting of polarimetric thermal facial signatures of 111 subjects is also introduced. Multiple experiments evaluated on different experimental protocols demonstrate that the proposed method achieves state-of-the-art performance. Code will be made available at https://github.com/hezhangsprinter.

arXiv commentNote that the extended dataset is available upon request. Researchers can contact Dr. Sean Hu from ARL at [email protected] to obtain the dataset

References in corpus (22)

• Adam: A Method for Stochastic Optimization
• Conditional Generative Adversarial Nets
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Densely Connected Convolutional Networks
• Improved Techniques for Training GANs
• Energy-based Generative Adversarial Network
• BEGAN: Boundary Equilibrium Generative Adversarial Networks
• Holistically-Nested Edge Detection
• Generative Image Inpainting with Contextual Attention
• A new face database simultaneously acquired in visible, near infrared and thermal spectrum
• Learning to Generate Images of Outdoor Scenes from Attributes and Semantic Layouts
• Multi-style Generative Network for Real-time Transfer
• Texture Synthesis with Spatial Generative Adversarial Networks
• StyleBank: An Explicit Representation for Neural Image Style Transfer
• Translating and Segmenting Multimodal Medical Volumes with Cycle- and Shape-Consistency Generative Adversarial Network
• Deep Perceptual Mapping for Thermal to Visible Face Recognition
• Jointly Optimize Data Augmentation and Network Training: Adversarial Data Augmentation in Human Pose Estimation
• Wasserstein CNN: Learning Invariant Features for NIR-VIS Face Recognition
• Reconstruction-Based Disentanglement for Pose-invariant Face Recognition
• Task Specific Adversarial Cost Function
• Deep Heterogeneous Feature Fusion for Template-Based Face Recognition
• Disentangled Variational Representation for Heterogeneous Face Recognition