Modeling emotion in complex stories: the Stanford Emotional Narratives Dataset

Abstract

Human emotions unfold over time, and more affective computing research has to prioritize capturing this crucial component of real-world affect. Modeling dynamic emotional stimuli requires solving the twin challenges of time-series modeling and of collecting high-quality time-series datasets. We begin by assessing the state-of-the-art in time-series emotion recognition, and we review contemporary time-series approaches in affective computing, including discriminative and generative models. We then introduce the first version of the Stanford Emotional Narratives Dataset (SENDv1): a set of rich, multimodal videos of self-paced, unscripted emotional narratives, annotated for emotional valence over time. The complex narratives and naturalistic expressions in this dataset provide a challenging test for contemporary time-series emotion recognition models. We demonstrate several baseline and state-of-the-art modeling approaches on the SEND, including a Long Short-Term Memory model and a multimodal Variational Recurrent Neural Network, which perform comparably to the human-benchmark. We end by discussing the implications for future research in time-series affective computing.

arXiv comment16 pages, 7 figures; accepted for publication at IEEE Transactions on Affective Computing

References in corpus (17)

• Scikit-learn: Machine Learning in Python
• Auto-Encoding Variational Bayes
• Neural Machine Translation by Jointly Learning to Align and Translate
• Sequence to Sequence Learning with Neural Networks
• Variational Inference: A Review for Statisticians
• AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild
• A Recurrent Latent Variable Model for Sequential Data
• Deep Affect Prediction in-the-wild: Aff-Wild Database and Challenge, Deep Architectures, and Beyond
• SEWA DB: A Rich Database for Audio-Visual Emotion and Sentiment Research in the Wild
• Learning Stochastic Recurrent Networks
• Multimodal Generative Models for Scalable Weakly-Supervised Learning
• Black box variational inference for state space models
• AVEC 2016 - Depression, Mood, and Emotion Recognition Workshop and Challenge
• Discovering Latent Network Structure in Point Process Data
• Modeling The Intensity Function Of Point Process Via Recurrent Neural Networks
• Applying Probabilistic Programming to Affective Computing
• Factorized Inference in Deep Markov Models for Incomplete Multimodal Time Series

Cited by in corpus (7)

• Emotion Intensity and its Control for Emotional Voice Conversion
• Hidden bawls, whispers, and yelps: can text be made to sound more than just its words?
• Context-Guided BERT for Targeted Aspect-Based Sentiment Analysis
• Artificial Intelligence for Emotion-Semantic Trending and People Emotion Detection During COVID-19 Social Isolation
• Attending to Emotional Narratives
• Understanding Emotion Valence is a Joint Deep Learning Task
• Structured Self-Attention Weights Encode Semantics in Sentiment Analysis