Words Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviors

Abstract

Humans convey their intentions through the usage of both verbal and nonverbal behaviors during face-to-face communication. Speaker intentions often vary dynamically depending on different nonverbal contexts, such as vocal patterns and facial expressions. As a result, when modeling human language, it is essential to not only consider the literal meaning of the words but also the nonverbal contexts in which these words appear. To better model human language, we first model expressive nonverbal representations by analyzing the fine-grained visual and acoustic patterns that occur during word segments. In addition, we seek to capture the dynamic nature of nonverbal intents by shifting word representations based on the accompanying nonverbal behaviors. To this end, we propose the Recurrent Attended Variation Embedding Network (RAVEN) that models the fine-grained structure of nonverbal subword sequences and dynamically shifts word representations based on nonverbal cues. Our proposed model achieves competitive performance on two publicly available datasets for multimodal sentiment analysis and emotion recognition. We also visualize the shifted word representations in different nonverbal contexts and summarize common patterns regarding multimodal variations of word representations.

arXiv commentAccepted by AAAI2019

References in corpus (9)

• A Survey on Multi-view Learning
• Character-Aware Neural Language Models
• Enriching Word Vectors with Subword Information
• Learning Factorized Multimodal Representations
• Dynamic Word Embeddings
• Probabilistic FastText for Multi-Sense Word Embeddings
• Select-Additive Learning: Improving Generalization in Multimodal Sentiment Analysis
• Seq2Seq2Sentiment: Multimodal Sequence to Sequence Models for Sentiment Analysis
• Multimodal Local-Global Ranking Fusion for Emotion Recognition

Cited by in corpus (5)

• UR-FUNNY: A Multimodal Language Dataset for Understanding Humor
• Factorized Multimodal Transformer for Multimodal Sequential Learning
• Learning Relationships between Text, Audio, and Video via Deep Canonical Correlation for Multimodal Language Analysis
• Integrating Multimodal Information in Large Pretrained Transformers
• Improving Multimodal fusion via Mutual Dependency Maximisation