Survey on Deep Multi-modal Data Analytics: Collaboration, Rivalry and Fusion
arXiv:2006.08159
Abstract
With the development of web technology, multi-modal or multi-view data has surged as a major stream for big data, where each modal/view encodes individual property of data objects. Often, different modalities are complementary to each other. Such fact motivated a lot of research attention on fusing the multi-modal feature spaces to comprehensively characterize the data objects. Most of the existing state-of-the-art focused on how to fuse the energy or information from multi-modal spaces to deliver a superior performance over their counterparts with single modal. Recently, deep neural networks have exhibited as a powerful architecture to well capture the nonlinear distribution of high-dimensional multimedia data, so naturally does for multi-modal data. Substantial empirical studies are carried out to demonstrate its advantages that are benefited from deep multi-modal methods, which can essentially deepen the fusion from multi-modal deep feature spaces. In this paper, we provide a substantial overview of the existing state-of-the-arts on the filed of multi-modal data analytics from shallow to deep spaces. Throughout this survey, we further indicate that the critical components for this field go to collaboration, adversarial competition and fusion over multi-modal spaces. Finally, we share our viewpoints regarding some future directions on this field.
Appearing at ACM TOMM, 26 pages
References in corpus (9)
- Conditional Generative Adversarial Nets
- Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks
- Multi-View Spectral Clustering via Structured Low-Rank Matrix Factorization
- Iterative Views Agreement: An Iterative Low-Rank based Structured Optimization Method to Multi-View Spectral Clustering
- Person Search with Natural Language Description
- Structured Generative Adversarial Networks
- Coupled CycleGAN: Unsupervised Hashing Network for Cross-Modal Retrieval
- Adversarial Cross-Modal Retrieval via Learning and Transferring Single-Modal Similarities
- Recovering Accurate Labeling Information from Partially Valid Data for Effective Multi-Label Learning