Survey on Deep Multi-modal Data Analytics: Collaboration, Rivalry and Fusion

Abstract

With the development of web technology, multi-modal or multi-view data has surged as a major stream for big data, where each modal/view encodes individual property of data objects. Often, different modalities are complementary to each other. Such fact motivated a lot of research attention on fusing the multi-modal feature spaces to comprehensively characterize the data objects. Most of the existing state-of-the-art focused on how to fuse the energy or information from multi-modal spaces to deliver a superior performance over their counterparts with single modal. Recently, deep neural networks have exhibited as a powerful architecture to well capture the nonlinear distribution of high-dimensional multimedia data, so naturally does for multi-modal data. Substantial empirical studies are carried out to demonstrate its advantages that are benefited from deep multi-modal methods, which can essentially deepen the fusion from multi-modal deep feature spaces. In this paper, we provide a substantial overview of the existing state-of-the-arts on the filed of multi-modal data analytics from shallow to deep spaces. Throughout this survey, we further indicate that the critical components for this field go to collaboration, adversarial competition and fusion over multi-modal spaces. Finally, we share our viewpoints regarding some future directions on this field.

Appearing at ACM TOMM, 26 pages

References in corpus (23)

• Conditional Generative Adversarial Nets
• Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks
• Improved Techniques for Training GANs
• InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets
• Adversarially Learned Inference
• Robust Face Recognition via Multimodal Deep Face Representation
• Multi-View Spectral Clustering via Structured Low-Rank Matrix Factorization
• Effective Multi-Query Expansions: Collaborative Deep Networks for Robust Landmark Retrieval
• Learning Deep Representations of Fine-grained Visual Descriptions
• Iterative Views Agreement: An Iterative Low-Rank based Structured Optimization Method to Multi-View Spectral Clustering
• A new face database simultaneously acquired in visible, near infrared and thermal spectrum
• Person Search with Natural Language Description
• CR-GAN: Learning Complete Representations for Multi-view Generation
• Multi-view Generative Adversarial Networks
• Structured Generative Adversarial Networks
• Load Balanced GANs for Multi-view Face Image Synthesis
• Modeling Text with Graph Convolutional Network for Cross-Modal Information Retrieval
• Coupled CycleGAN: Unsupervised Hashing Network for Cross-Modal Retrieval
• CCL: Cross-modal Correlation Learning with Multi-grained Fusion by Hierarchical Network
• Multi-View Frame Reconstruction with Conditional GAN
• MEGAN: A Generative Adversarial Network for Multi-View Network Embedding
• Recovering Accurate Labeling Information from Partially Valid Data for Effective Multi-Label Learning
• Adversarial Cross-Modal Retrieval via Learning and Transferring Single-Modal Similarities

Cited by in corpus (4)

• From CNNs to Transformers in Multimodal Human Action Recognition: A Survey
• Structured Graph Learning for Scalable Subspace Clustering: From Single-view to Multi-view
• Exploring Image Enhancement for Salient Object Detection in Low Light Images
• Deep Adversarial Inconsistent Cognitive Sampling for Multi-view Progressive Subspace Clustering