Multimodal Intelligence: Representation Learning, Information Fusion, and Applications

Abstract

Deep learning methods have revolutionized speech recognition, image recognition, and natural language processing since 2010. Each of these tasks involves a single modality in their input signals. However, many applications in the artificial intelligence field involve multiple modalities. Therefore, it is of broad interest to study the more difficult and complex problem of modeling and learning across multiple modalities. In this paper, we provide a technical review of available models and learning methods for multimodal intelligence. The main focus of this review is the combination of vision and natural language modalities, which has become an important topic in both the computer vision and natural language processing research communities. This review provides a comprehensive analysis of recent works on multimodal deep learning from three perspectives: learning multimodal representations, fusing multimodal signals at various levels, and multimodal applications. Regarding multimodal representation learning, we review the key concepts of embedding, which unify multimodal signals into a single vector space and thereby enable cross-modality signal processing. We also review the properties of many types of embeddings that are constructed and learned for general downstream tasks. Regarding multimodal fusion, this review focuses on special architectures for the integration of representations of unimodal signals for a particular task. Regarding applications, selected areas of a broad interest in the current literature are covered, including image-to-text caption generation, text-to-image generation, and visual question answering. We believe that this review will facilitate future studies in the emerging field of multimodal intelligence for related communities.

References in corpus (69)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Efficient Estimation of Word Representations in Vector Space
• Deep Learning in Neural Networks: An Overview
• Distributed Representations of Words and Phrases and their Compositionality
• Neural Machine Translation by Jointly Learning to Align and Translate
• Sequence to Sequence Learning with Neural Networks
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Conditional Generative Adversarial Nets
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Two-Stream Convolutional Networks for Action Recognition in Videos
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Neural Architecture Search with Reinforcement Learning
• WaveNet: A Generative Model for Raw Audio
• VoxCeleb2: Deep Speaker Recognition
• Generative Adversarial Text to Image Synthesis
• Conditional Image Synthesis With Auxiliary Classifier GANs
• ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
• Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks
• A Structured Self-attentive Sentence Embedding
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• VisualBERT: A Simple and Performant Baseline for Vision and Language
• Hierarchical Question-Image Co-Attention for Visual Question Answering
• VQA: Visual Question Answering
• Zero-Shot Learning Through Cross-Modal Transfer
• Deep Audio-Visual Speech Recognition
• VL-BERT: Pre-training of Generic Visual-Linguistic Representations
• Deep Fragment Embeddings for Bidirectional Image Sentence Mapping
• Skip-Thought Vectors
• Learning Representations by Maximizing Mutual Information Across Views
• Dynamic Memory Networks for Visual and Textual Question Answering
• Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)
• Deep Sentence Embedding Using Long Short-Term Memory Networks: Analysis and Application to Information Retrieval
• Looking to Listen at the Cocktail Party: A Speaker-Independent Audio-Visual Model for Speech Separation
• Beyond Bilinear: Generalized Multimodal Factorized High-order Pooling for Visual Question Answering
• Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis
• Exploring Models and Data for Image Question Answering
• Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding
• Quasi-Recurrent Neural Networks
• Simple Baseline for Visual Question Answering
• Multimodal Residual Learning for Visual QA
• Learning Factorized Multimodal Representations
• Learning What and Where to Draw
• Transparency by Design: Closing the Gap Between Performance and Interpretability in Visual Reasoning
• RUBi: Reducing Unimodal Biases in Visual Question Answering
• Learning to Count Objects in Natural Images for Visual Question Answering
• Ask Your Neurons: A Neural-based Approach to Answering Questions about Images
• Hadamard Product for Low-rank Bilinear Pooling
• The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision
• Self-Monitoring Navigation Agent via Auxiliary Progress Estimation
• Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training
• Audio-Visual Speaker Diarization Based on Spatiotemporal Bayesian Fusion
• TAC-GAN - Text Conditioned Auxiliary Classifier Generative Adversarial Network
• ChatPainter: Improving Text to Image Generation using Dialogue
• From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Following
• Text-Adaptive Generative Adversarial Networks: Manipulating Images with Natural Language
• Gated Multimodal Units for Information Fusion
• EmoBed: Strengthening Monomodal Emotion Recognition via Training with Crossmodal Emotion Embeddings
• High-Order Attention Models for Visual Question Answering
• Rich Image Captioning in the Wild
• Probabilistic Neural-symbolic Models for Interpretable Visual Question Answering
• Creativity: Generating Diverse Questions using Variational Autoencoders
• Neural Language Modeling with Visual Features
• Active Learning for Visual Question Answering: An Empirical Study
• OmniNet: A unified architecture for multi-modal multi-task learning
• Are You Looking? Grounding to Multiple Modalities in Vision-and-Language Navigation
• Efficient Progressive Neural Architecture Search
• Tactical Rewind: Self-Correction via Backtracking in Vision-and-Language Navigation
• Deep Bayesian Active Learning for Multiple Correct Outputs

Cited by in corpus (30)

• Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directions
• Imbalanced Big Data Oversampling: Taxonomy, Algorithms, Software, Guidelines and Future Directions
• A survey of multimodal deep generative models
• A Systematic Review of Intermediate Fusion in Multimodal Deep Learning for Biomedical Applications
• Internet of Intelligence: A Survey on the Enabling Technologies, Applications, and Challenges
• From Google Gemini to OpenAI Q* (Q-Star): A Survey of Reshaping the Generative Artificial Intelligence (AI) Research Landscape
• Automated scholarly paper review: Concepts, technologies, and challenges
• Graph-Based Intercategory and Intermodality Network for Multilabel Classification and Melanoma Diagnosis of Skin Lesions in Dermoscopy and Clinical Images
• Multi-Modal Deep Learning for Credit Rating Prediction Using Text and Numerical Data Streams
• Bi-Level Graph Structure Learning for Next POI Recommendation
• Object Detection with Multimodal Large Vision-Language Models: An In-depth Review
• Ubi-SleepNet: Advanced Multimodal Fusion Techniques for Three-stage Sleep Classification Using Ubiquitous Sensing
• Recent Advances and Trends in Multimodal Deep Learning: A Review
• Combination of Deep Speaker Embeddings for Diarisation
• Detection of Illicit Drug Trafficking Events on Instagram: A Deep Multimodal Multilabel Learning Approach
• How to find a good image-text embedding for remote sensing visual question answering?
• Relating by Contrasting: A Data-efficient Framework for Multimodal Generative Models
• Literature review on vulnerability detection using NLP technology
• Cross-modal feature fusion for robust point cloud registration with ambiguous geometry
• ScaleVLAD: Improving Multimodal Sentiment Analysis via Multi-Scale Fusion of Locally Descriptors
• Distribution-based Emotion Recognition in Conversation
• Automatic Fused Multimodal Deep Learning for Plant Identification
• Towards Multimodal Graph Large Language Model
• Exploring Deep Learning for Joint Audio-Visual Lip Biometrics
• Towards Better Multi-modal Keyphrase Generation via Visual Entity Enhancement and Multi-granularity Image Noise Filtering
• Trying Bilinear Pooling in Video-QA
• Early-fusion Based Pulsar Identification with Smart Under-sampling
• Reflections on "Can AI Understand Our Universe?"
• What is Multimodality?
• Enhancing Modality Representation and Alignment for Multimodal Cold-start Active Learning