Image-Text Multi-Modal Representation Learning by Adversarial Backpropagation

Abstract

We present novel method for image-text multi-modal representation learning. In our knowledge, this work is the first approach of applying adversarial learning concept to multi-modal learning and not exploiting image-text pair information to learn multi-modal feature. We only use category information in contrast with most previous methods using image-text pair information for multi-modal embedding. In this paper, we show that multi-modal feature can be achieved without image-text pair information and our method makes more similar distribution with image and text in multi-modal feature space than other methods which use image-text pair information. And we show our multi-modal feature has universal semantic information, even though it was trained for category prediction. Our model is end-to-end backpropagation, intuitive and easily extended to other multi-modal learning work.

8 pages, 5 figures

Cited by in corpus (9)

• Adversarial Multi-task Learning for Text Classification
• MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis
• Content-Based Video-Music Retrieval Using Soft Intra-Modal Structure Constraint
• A New Evaluation Protocol and Benchmarking Results for Extendable Cross-media Retrieval
• Revisiting Cross Modal Retrieval
• JTAV: Jointly Learning Social Media Content Representation by Fusing Textual, Acoustic, and Visual Features
• Multimodal Prediction based on Graph Representations
• Self-Augmented Multi-Modal Feature Embedding
• Learning Inward Scaled Hypersphere Embedding: Exploring Projections in Higher Dimensions