Multi-modal Deep Analysis for Multimedia
arXiv:1910.04964 · doi:10.1109/TCSVT.2019.2940647
Abstract
With the rapid development of Internet and multimedia services in the past decade, a huge amount of user-generated and service provider-generated multimedia data become available. These data are heterogeneous and multi-modal in nature, imposing great challenges for processing and analyzing them. Multi-modal data consist of a mixture of various types of data from different modalities such as texts, images, videos, audios etc. In this article, we present a deep and comprehensive overview for multi-modal analysis in multimedia. We introduce two scientific research problems, data-driven correlational representation and knowledge-guided fusion for multimedia analysis. To address the two scientific problems, we investigate them from the following aspects: 1) multi-modal correlational representation: multi-modal fusion of data across different modalities, and 2) multi-modal data and knowledge fusion: multi-modal fusion of data with domain knowledge. More specifically, on data-driven correlational representation, we highlight three important categories of methods, such as multi-modal deep representation, multi-modal transfer learning, and multi-modal hashing. On knowledge-guided fusion, we discuss the approaches for fusing knowledge with data and four exemplar applications that require various kinds of domain knowledge, including multi-modal visual question answering, multi-modal video summarization, multi-modal visual pattern mining and multi-modal recommendation. Finally, we bring forward our insights and future research directions.
25 pages, 39 figures, IEEE Transactions on Circuits and Systems for Video Technology
References in corpus (35)
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- Auto-Encoding Variational Bayes
- Distilling the Knowledge in a Neural Network
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Two-Stream Convolutional Networks for Action Recognition in Videos
- How transferable are features in deep neural networks?
- Learning Transferable Features with Deep Adaptation Networks
- Unsupervised Domain Adaptation by Backpropagation
- An Overview of Multi-Task Learning in Deep Neural Networks
- Deep Domain Confusion: Maximizing for Domain Invariance
- Generative Adversarial Text to Image Synthesis
- Frustratingly Easy Domain Adaptation
- Hierarchical Question-Image Co-Attention for Visual Question Answering
- A Survey on Multi-view Learning
- Deep Fragment Embeddings for Bidirectional Image Sentence Mapping
- Dynamic Memory Networks for Visual and Textual Question Answering
- A Survey on Multi-Task Learning
- Equivalence of distance-based and RKHS-based statistics in hypothesis testing
- Domain Separation Networks
- Marginalized Denoising Autoencoders for Domain Adaptation
- Robust Face Recognition via Multimodal Deep Face Representation
- Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
- Exploring Models and Data for Image Question Answering
- Multimodal Residual Learning for Visual QA
- Cycle-Consistent Deep Generative Hashing for Cross-Modal Retrieval
- Transparency by Design: Closing the Gap Between Performance and Interpretability in Visual Reasoning
- Learning Conditioned Graph Structures for Interpretable Visual Question Answering
- Joint Multimodal Learning with Deep Generative Models
- Explicit Knowledge-based Reasoning for Visual Question Answering
- Incorporating External Knowledge to Answer Open-Domain Visual Questions with Dynamic Memory Networks
- Shared Representation Learning for Heterogeneous Face Recognition
- Multimodal Transfer Deep Learning with Applications in Audio-Visual Recognition
- PatternNet: Visual Pattern Mining with Deep Neural Network
- Selective Transfer Learning for Cross Domain Recommendation
- Better and Faster: Knowledge Transfer from Multiple Self-supervised Learning Tasks via Graph Distillation for Video Classification