CM-GANs: Cross-modal Generative Adversarial Networks for Common Representation Learning

Abstract

It is known that the inconsistent distribution and representation of different modalities, such as image and text, cause the heterogeneity gap that makes it challenging to correlate such heterogeneous data. Generative adversarial networks (GANs) have shown its strong ability of modeling data distribution and learning discriminative representation, existing GANs-based works mainly focus on generative problem to generate new data. We have different goal, aim to correlate heterogeneous data, by utilizing the power of GANs to model cross-modal joint distribution. Thus, we propose Cross-modal GANs to learn discriminative common representation for bridging heterogeneity gap. The main contributions are: (1) Cross-modal GANs architecture is proposed to model joint distribution over data of different modalities. The inter-modality and intra-modality correlation can be explored simultaneously in generative and discriminative models. Both of them beat each other to promote cross-modal correlation learning. (2) Cross-modal convolutional autoencoders with weight-sharing constraint are proposed to form generative model. They can not only exploit cross-modal correlation for learning common representation, but also preserve reconstruction information for capturing semantic consistency within each modality. (3) Cross-modal adversarial mechanism is proposed, which utilizes two kinds of discriminative models to simultaneously conduct intra-modality and inter-modality discrimination. They can mutually boost to make common representation more discriminative by adversarial training process. To the best of our knowledge, our proposed CM-GANs approach is the first to utilize GANs to perform cross-modal common representation learning. Experiments are conducted to verify the performance of our proposed approach on cross-modal retrieval paradigm, compared with 10 methods on 3 cross-modal datasets.

References in corpus (7)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Conditional Generative Adversarial Nets
• Generative Adversarial Text to Image Synthesis
• Conditional Image Synthesis With Auxiliary Classifier GANs
• Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks
• Unsupervised Learning for Physical Interaction through Video Prediction
• Perceptual Generative Adversarial Networks for Small Object Detection

Cited by in corpus (12)

• Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directions
• Learning Dual Semantic Relations with Graph Attention for Image-Text Matching
• Stacked Cross Attention for Image-Text Matching
• Dropout-GAN: Learning from a Dynamic Ensemble of Discriminators
• Learning Visual Relation Priors for Image-Text Matching and Image Captioning with Neural Scene Graph Generators
• Learning Cross-Modal Embeddings with Adversarial Networks for Cooking Recipes and Food Images
• Cross-Modal Food Retrieval: Learning a Joint Embedding of Food Images and Recipes with Semantic Consistency and Attention Mechanism
• PM-GANs: Discriminative Representation Learning for Action Recognition Using Partial-modalities
• Adversarial Cross-Modal Retrieval via Learning and Transferring Single-Modal Similarities
• New Ideas and Trends in Deep Multimodal Content Understanding: A Review
• Multi-Modal Retrieval using Graph Neural Networks
• Learning to Clean: A GAN Perspective