Memory Fusion Network for Multi-view Sequential Learning

Abstract

Multi-view sequential learning is a fundamental problem in machine learning dealing with multi-view sequences. In a multi-view sequence, there exists two forms of interactions between different views: view-specific interactions and cross-view interactions. In this paper, we present a new neural architecture for multi-view sequential learning called the Memory Fusion Network (MFN) that explicitly accounts for both interactions in a neural architecture and continuously models them through time. The first component of the MFN is called the System of LSTMs, where view-specific interactions are learned in isolation through assigning an LSTM function to each view. The cross-view interactions are then identified using a special attention mechanism called the Delta-memory Attention Network (DMAN) and summarized through time with a Multi-view Gated Memory. Through extensive experimentation, MFN is compared to various proposed approaches for multi-view sequential learning on multiple publicly available benchmark datasets. MFN outperforms all the existing multi-view approaches. Furthermore, MFN outperforms all current state-of-the-art models, setting new state-of-the-art results for these multi-view datasets.

arXiv commentAAAI 2018 Oral Presentation

Cited by in corpus (36)

• Multilogue-Net: A Context Aware RNN for Multi-modal Emotion Detection and Sentiment Analysis in Conversation
• Multi-modal Sarcasm Detection and Humor Classification in Code-mixed Conversations
• Attention-based Audio-Visual Fusion for Robust Automatic Speech Recognition
• Semi-supervised Multi-modal Emotion Recognition with Cross-Modal Distribution Matching
• MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis
• Graph Capsule Aggregation for Unaligned Multimodal Sequences
• Factorized Multimodal Transformer for Multimodal Sequential Learning
• Analyzing Unaligned Multimodal Sequence via Graph Convolution and Graph Pooling Fusion
• Learning Relationships between Text, Audio, and Video via Deep Canonical Correlation for Multimodal Language Analysis
• Modality to Modality Translation: An Adversarial Representation Learning and Graph Fusion Network for Multimodal Fusion
• Improving Multimodal Accuracy Through Modality Pre-training and Attention
• Emotions Don't Lie: An Audio-Visual Deepfake Detection Method Using Affective Cues
• Weakly-supervised Multi-task Learning for Multimodal Affect Recognition
• Collaborative Attention Mechanism for Multi-View Action Recognition
• ScaleVLAD: Improving Multimodal Sentiment Analysis via Multi-Scale Fusion of Locally Descriptors
• M2Lens: Visualizing and Explaining Multimodal Models for Sentiment Analysis
• Correlative Channel-Aware Fusion for Multi-View Time Series Classification
• Audio-Visual Event Localization via Recursive Fusion by Joint Co-Attention
• HetMAML: Task-Heterogeneous Model-Agnostic Meta-Learning for Few-Shot Learning Across Modalities
• Video Sentiment Analysis with Bimodal Information-augmented Multi-Head Attention
• Variational Fusion for Multimodal Sentiment Analysis
• TEASEL: A Transformer-Based Speech-Prefixed Language Model
• Unsupervised Multimodal Language Representations using Convolutional Autoencoders
• Deep-HOSeq: Deep Higher Order Sequence Fusion for Multimodal Sentiment Analysis
• Memory and attention in deep learning
• Neural Dependency Coding inspired Multimodal Fusion
• Multimodal Data Fusion based on the Global Workspace Theory
• Quantum-inspired Multimodal Fusion for Video Sentiment Analysis
• Attending to Emotional Narratives
• Multimodal Embeddings from Language Models
• A Multimodal Sentiment Dataset for Video Recommendation
• Sentence-level Online Handwritten Chinese Character Recognition
• Memory based fusion for multi-modal deep learning
• Improving Multimodal fusion via Mutual Dependency Maximisation
• Multi-Modal Sarcasm Detection Based on Contrastive Attention Mechanism
• Hierachical Delta-Attention Method for Multimodal Fusion