HashGAN:Attention-aware Deep Adversarial Hashing for Cross Modal Retrieval

Abstract

As the rapid growth of multi-modal data, hashing methods for cross-modal retrieval have received considerable attention. Deep-networks-based cross-modal hashing methods are appealing as they can integrate feature learning and hash coding into end-to-end trainable frameworks. However, it is still challenging to find content similarities between different modalities of data due to the heterogeneity gap. To further address this problem, we propose an adversarial hashing network with attention mechanism to enhance the measurement of content similarities by selectively focusing on informative parts of multi-modal data. The proposed new adversarial network, HashGAN, consists of three building blocks: 1) the feature learning module to obtain feature representations, 2) the generative attention module to generate an attention mask, which is used to obtain the attended (foreground) and the unattended (background) feature representations, 3) the discriminative hash coding module to learn hash functions that preserve the similarities between different modalities. In our framework, the generative module and the discriminative module are trained in an adversarial way: the generator is learned to make the discriminator cannot preserve the similarities of multi-modal data w.r.t. the background feature representations, while the discriminator aims to preserve the similarities of multi-modal data w.r.t. both the foreground and the background feature representations. Extensive evaluations on several benchmark datasets demonstrate that the proposed HashGAN brings substantial improvements over other state-of-the-art cross-modal hashing methods.

arXiv comment10 pages, 8 figures, 3 tables

References in corpus (11)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Conditional Generative Adversarial Nets
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• Simultaneous Feature Learning and Hash Coding with Deep Neural Networks
• Return of the Devil in the Details: Delving Deep into Convolutional Nets
• Action Recognition using Visual Attention
• A Comprehensive Survey on Cross-modal Retrieval
• Deep Cross-Modal Hashing

Cited by in corpus (2)

• Deep Learning Techniques for Future Intelligent Cross-Media Retrieval
• Learning Shared Semantic Space with Correlation Alignment for Cross-modal Event Retrieval