Recent Advances and Trends in Multimodal Deep Learning: A Review
arXiv:2105.11087
Abstract
Deep Learning has implemented a wide range of applications and has become increasingly popular in recent years. The goal of multimodal deep learning is to create models that can process and link information using various modalities. Despite the extensive development made for unimodal learning, it still cannot cover all the aspects of human learning. Multimodal learning helps to understand and analyze better when various senses are engaged in the processing of information. This paper focuses on multiple types of modalities, i.e., image, video, text, audio, body gestures, facial expressions, and physiological signals. Detailed analysis of past and current baseline approaches and an in-depth study of recent advancements in multimodal deep learning applications has been provided. A fine-grained taxonomy of various multimodal deep learning applications is proposed, elaborating on different applications in more depth. Architectures and datasets used in these applications are also discussed, along with their evaluation metrics. Last, main issues are highlighted separately for each domain along with their possible future research directions.
References in corpus (8)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- Microsoft COCO Captions: Data Collection and Evaluation Server
- Deep Voice: Real-time Neural Text-to-Speech
- A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Input
- Using Descriptive Video Services to Create a Large Data Source for Video Annotation Research
- CPTR: Full Transformer Network for Image Captioning
- What Does a TextCNN Learn?