Recent Advances and Trends in Multimodal Deep Learning: A Review

Abstract

Deep Learning has implemented a wide range of applications and has become increasingly popular in recent years. The goal of multimodal deep learning is to create models that can process and link information using various modalities. Despite the extensive development made for unimodal learning, it still cannot cover all the aspects of human learning. Multimodal learning helps to understand and analyze better when various senses are engaged in the processing of information. This paper focuses on multiple types of modalities, i.e., image, video, text, audio, body gestures, facial expressions, and physiological signals. Detailed analysis of past and current baseline approaches and an in-depth study of recent advancements in multimodal deep learning applications has been provided. A fine-grained taxonomy of various multimodal deep learning applications is proposed, elaborating on different applications in more depth. Architectures and datasets used in these applications are also discussed, along with their evaluation metrics. Last, main issues are highlighted separately for each domain along with their possible future research directions.

References in corpus (24)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Microsoft COCO Captions: Data Collection and Evaluation Server
• VQA: Visual Question Answering
• Multimodal Intelligence: Representation Learning, Information Fusion, and Applications
• Deep Voice: Real-time Neural Text-to-Speech
• Exploring Models and Data for Image Question Answering
• RSVQA: Visual Question Answering for Remote Sensing Data
• A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Input
• Coherent Multi-Sentence Video Description with Variable Level of Detail
• Using Descriptive Video Services to Create a Large Data Source for Video Annotation Research
• Tacotron: Towards End-to-End Speech Synthesis
• Cross-modal Knowledge Reasoning for Knowledge-based Visual Question Answering
• CPTR: Full Transformer Network for Image Captioning
• Deep Voice 3: Scaling Text-to-Speech with Convolutional Sequence Learning
• Multi-Modal Emotion recognition on IEMOCAP Dataset using Deep Learning
• HEU Emotion: A Large-scale Database for Multi-modal Emotion Recognition in the Wild
• Attentive Explanations: Justifying Decisions and Pointing to the Evidence
• Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
• Parallel Tacotron: Non-Autoregressive and Controllable TTS
• Controlling for Confounders in Multimodal Emotion Classification via Adversarial Learning
• Multimodal Sparse Coding for Event Detection
• What Does a TextCNN Learn?
• Video captioning with stacked attention and semantic hard pull