Attention Control with Metric Learning Alignment for Image Set-based Recognition

Abstract

This paper considers the problem of image set-based face verification and identification. Unlike traditional single sample (an image or a video) setting, this situation assumes the availability of a set of heterogeneous collection of orderless images and videos. The samples can be taken at different check points, different identity documents $etc$. The importance of each image is usually considered either equal or based on a quality assessment of that image independent of other images and/or videos in that image set. How to model the relationship of orderless images within a set remains a challenge. We address this problem by formulating it as a Markov Decision Process (MDP) in a latent space. Specifically, we first propose a dependency-aware attention control (DAC) network, which uses actor-critic reinforcement learning for attention decision of each image to exploit the correlations among the unordered images. An off-policy experience replay is introduced to speed up the learning process. Moreover, the DAC is combined with a temporal model for videos using divide and conquer strategies. We also introduce a pose-guided representation (PGR) scheme that can further boost the performance at extreme poses. We propose a parameter-free PGR without the need for training as well as a novel metric learning-based PGR for pose alignment without the need for pose detection in testing stage. Extensive evaluations on IJB-A/B/C, YTF, Celebrity-1000 datasets demonstrate that our method outperforms many state-of-art approaches on the set-based as well as video-based face recognition databases.

Accepted to IEEE T-IFS (Extension of ECCV 2018 paper: Dependency-aware Attention Control for Unconstrained Face Recognition with Image Sets). arXiv admin note: substantial text overlap with arXiv:1907.03030; text overlap with arXiv:1707.00130 by other authors

References in corpus (15)

• Continuous control with deep reinforcement learning
• Recurrent Models of Visual Attention
• Deep Reinforcement Learning: An Overview
• Trunk-Branch Ensemble Convolutional Neural Networks for Video-based Face Recognition
• SphereFace: Deep Hypersphere Embedding for Face Recognition
• Revisiting Temporal Modeling for Video-based Person ReID
• Multicolumn Networks for Face Recognition
• Face Search at Scale: 80 Million Gallery
• Towards End-to-End Speech Recognition with Deep Convolutional Neural Networks
• Do We Really Need to Collect Millions of Faces for Effective Face Recognition?
• Quality Aware Network for Set to Set Recognition
• Learning Policies for Adaptive Tracking with Deep Feature Cascades
• Conservative Wasserstein Training for Pose Estimation
• Deep Reinforcement Learning Attention Selection for Person Re-Identification
• Multi-shot Pedestrian Re-identification via Sequential Decision Making

Cited by in corpus (3)

• Disentanglement for Discriminative Visual Recognition
• Energy-constrained Self-training for Unsupervised Domain Adaptation
• Identity-aware Facial Expression Recognition in Compressed Video