New Ideas and Trends in Deep Multimodal Content Understanding: A Review
arXiv:2010.08189
Abstract
The focus of this survey is on the analysis of two modalities of multimodal deep learning: image and text. Unlike classic reviews of deep learning where monomodal image classifiers such as VGG, ResNet and Inception module are central topics, this paper will examine recent multimodal deep models and structures, including auto-encoders, generative adversarial nets and their variants. These models go beyond the simple image classifiers in which they can do uni-directional (e.g. image captioning, image generation) and bi-directional (e.g. cross-modal retrieval, visual question answering) multimodal tasks. Besides, we analyze two aspects of the challenge in terms of better content understanding in deep multimodal applications. We then introduce current ideas and trends in deep multimodal feature learning, such as feature embedding approaches and objective function design, which are crucial in overcoming the aforementioned challenges. Finally, we include several promising directions for future research.
arXiv commentAccepted by Neurocomputing
References in corpus (33)
- Semi-Supervised Classification with Graph Convolutional Networks
- Self-Attention Generative Adversarial Networks
- Generative Adversarial Text to Image Synthesis
- Conditional Image Synthesis With Auxiliary Classifier GANs
- Graph Neural Networks: A Review of Methods and Applications
- Recent Trends in Deep Learning Based Natural Language Processing
- Dynamic Memory Networks for Visual and Textual Question Answering
- Beyond Bilinear: Generalized Multimodal Factorized High-order Pooling for Visual Question Answering
- Improved Image Captioning via Policy Gradient optimization of SPIDEr
- Visual Question Answering: Datasets, Algorithms, and Future Challenges
- Learning What and Where to Draw
- An Introduction to Image Synthesis with Generative Adversarial Nets
- Compositional Attention Networks for Machine Reasoning
- Context-Aware Visual Policy Network for Sequence-Level Image Captioning
- Binary Generative Adversarial Networks for Image Retrieval
- A New Benchmark and Approach for Fine-grained Cross-media Retrieval
- Focal Visual-Text Attention for Visual Question Answering
- Generating Images from Captions with Attention
- See, Hear, and Read: Deep Aligned Representations
- CM-GANs: Cross-modal Generative Adversarial Networks for Common Representation Learning
- Relation-Aware Graph Attention Network for Visual Question Answering
- Exploring Human-like Attention Supervision in Visual Question Answering
- Multi-step Reasoning via Recurrent Dual Attention for Visual Dialog
- Auto-Encoding Scene Graphs for Image Captioning
- Partially-Supervised Image Captioning
- Learning to Guide Decoding for Image Captioning
- Text-to-Image-to-Text Translation using Cycle Consistent Adversarial Networks
- Attention on Attention: Architectures for Visual Question Answering (VQA)
- Dynamic Fusion with Intra- and Inter- Modality Attention Flow for Visual Question Answering
- GILT: Generating Images from Long Text
- Gated Hierarchical Attention for Image Captioning
- Dual Recurrent Attention Units for Visual Question Answering
- Cross-media Multi-level Alignment with Relation Attention Network