Factorized Multimodal Transformer for Multimodal Sequential Learning

Abstract

The complex world around us is inherently multimodal and sequential (continuous). Information is scattered across different modalities and requires multiple continuous sensors to be captured. As machine learning leaps towards better generalization to real world, multimodal sequential learning becomes a fundamental research area. Arguably, modeling arbitrarily distributed spatio-temporal dynamics within and across modalities is the biggest challenge in this research area. In this paper, we present a new transformer model, called the Factorized Multimodal Transformer (FMT) for multimodal sequential learning. FMT inherently models the intramodal and intermodal (involving two or more modalities) dynamics within its multimodal input in a factorized manner. The proposed factorization allows for increasing the number of self-attentions to better model the multimodal phenomena at hand; without encountering difficulties during training (e.g. overfitting) even on relatively low-resource setups. All the attention mechanisms within FMT have a full time-domain receptive field which allows them to asynchronously capture long-range multimodal dynamics. In our experiments we focus on datasets that contain the three commonly studied modalities of language, vision and acoustic. We perform a wide range of experiments, spanning across 3 well-studied datasets and 21 distinct labels. FMT shows superior performance over previously proposed models, setting new state of the art in the studied datasets.

References in corpus (12)

• Adam: A Method for Stochastic Optimization
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• A Survey on Multi-view Learning
• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videos
• Hadamard Product for Low-rank Bilinear Pooling
• Memory Fusion Network for Multi-view Sequential Learning
• Learning to Reason: End-to-End Module Networks for Visual Question Answering
• Efficient Low-rank Multimodal Fusion with Modality-Specific Factors
• Compact Bilinear Pooling
• Words Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviors

Cited by in corpus (8)

• Efficient Multimodal Transformer with Dual-Level Feature Restoration for Robust Multimodal Sentiment Analysis
• Graph Capsule Aggregation for Unaligned Multimodal Sequences
• ScaleVLAD: Improving Multimodal Sentiment Analysis via Multi-Scale Fusion of Locally Descriptors
• Multi-modal Attention for Speech Emotion Recognition
• Video Sentiment Analysis with Bimodal Information-augmented Multi-Head Attention
• TEASEL: A Transformer-Based Speech-Prefixed Language Model
• Neural Dependency Coding inspired Multimodal Fusion
• Improving Multimodal fusion via Mutual Dependency Maximisation