MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videos

Abstract

People are sharing their opinions, stories and reviews through online video sharing websites every day. Studying sentiment and subjectivity in these opinion videos is experiencing a growing attention from academia and industry. While sentiment analysis has been successful for text, it is an understudied research question for videos and multimedia content. The biggest setbacks for studies in this direction are lack of a proper dataset, methodology, baselines and statistical analysis of how information from different modality sources relate to each other. This paper introduces to the scientific community the first opinion-level annotated corpus of sentiment and subjectivity analysis in online videos called Multimodal Opinion-level Sentiment Intensity dataset (MOSI). The dataset is rigorously annotated with labels for subjectivity, sentiment intensity, per-frame and per-opinion annotated visual features, and per-milliseconds annotated audio features. Furthermore, we present baselines for future studies in this direction as well as a new multimodal fusion approach that jointly models spoken words and visual gestures.

Accepted as Journal Publication in IEEE Intelligent Systems

Cited by in corpus (44)

• Recent Trends in Deep Learning Based Personality Detection
• A Deeper Look into Sarcastic Tweets Using Deep Convolutional Neural Networks
• Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directions
• Exploiting BERT For Multimodal Target Sentiment Classification Through Input Space Translation
• UR-FUNNY: A Multimodal Language Dataset for Understanding Humor
• Multilogue-Net: A Context Aware RNN for Multi-modal Emotion Detection and Sentiment Analysis in Conversation
• Efficient Low-rank Multimodal Fusion with Modality-Specific Factors
• Emotional Speech-Driven Animation with Content-Emotion Disentanglement
• Complementary Fusion of Multi-Features and Multi-Modalities in Sentiment Analysis
• Trustworthy Multimodal Fusion for Sentiment Analysis in Ordinal Sentiment Space
• MSAF: Multimodal Split Attention Fusion
• Factorized Multimodal Transformer for Multimodal Sequential Learning
• Graph Capsule Aggregation for Unaligned Multimodal Sequences
• Analyzing Unaligned Multimodal Sequence via Graph Convolution and Graph Pooling Fusion
• Learning Relationships between Text, Audio, and Video via Deep Canonical Correlation for Multimodal Language Analysis
• Multimodal Learning for Multi-Omics: A Survey
• MultiBench: Multiscale Benchmarks for Multimodal Representation Learning
• Hybrid Multimodal Fusion for Humor Detection
• Combating Human Trafficking with Deep Multimodal Models
• Weakly-supervised Multi-task Learning for Multimodal Affect Recognition
• SMIL: Multimodal Learning with Severely Missing Modality
• From the Token to the Review: A Hierarchical Multimodal approach to Opinion Mining
• Multi-modal Sentiment Analysis using Deep Canonical Correlation Analysis
• Multimodal Affective Analysis Using Hierarchical Attention Strategy with Word-Level Alignment
• Learning Robust Heterogeneous Signal Features from Parallel Neural Network for Audio Sentiment Analysis
• Integrating Multimodal Information in Large Pretrained Transformers
• Quantum Cognitively Motivated Decision Fusion for Video Sentiment Analysis
• WildMix Dataset and Spectro-Temporal Transformer Model for Monoaural Audio Source Separation
• Multimodal Sentiment Analysis with Word-Level Fusion and Reinforcement Learning
• Multimodal End-to-End Sparse Model for Emotion Recognition
• Video Sentiment Analysis with Bimodal Information-augmented Multi-Head Attention
• Team Neuro at SemEval-2020 Task 8: Multi-Modal Fine Grain Emotion Classification of Memes using Multitask Learning
• TCGM: An Information-Theoretic Framework for Semi-Supervised Multi-Modality Learning
• TEASEL: A Transformer-Based Speech-Prefixed Language Model
• KT-Speech-Crawler: Automatic Dataset Construction for Speech Recognition from YouTube Videos
• A Multimodal Sentiment Dataset for Video Recommendation
• MUSER: MUltimodal Stress Detection using Emotion Recognition as an Auxiliary Task
• Multimodal End-to-End Group Emotion Recognition using Cross-Modal Attention
• TransModality: An End2End Fusion Method with Transformer for Multimodal Sentiment Analysis
• Quantum-inspired Multimodal Fusion for Video Sentiment Analysis
• Which is Making the Contribution: Modulating Unimodal and Cross-modal Dynamics for Multimodal Sentiment Analysis
• Sequential Late Fusion Technique for Multi-modal Sentiment Analysis
• Highlight Timestamp Detection Model for Comedy Videos via Multimodal Sentiment Analysis
• Modulated Fusion using Transformer for Linguistic-Acoustic Emotion Recognition