UR-FUNNY: A Multimodal Language Dataset for Understanding Humor

Abstract

Humor is a unique and creative communicative behavior displayed during social interactions. It is produced in a multimodal manner, through the usage of words (text), gestures (vision) and prosodic cues (acoustic). Understanding humor from these three modalities falls within boundaries of multimodal language; a recent research trend in natural language processing that models natural language as it happens in face-to-face communication. Although humor detection is an established research area in NLP, in a multimodal context it is an understudied area. This paper presents a diverse multimodal dataset, called UR-FUNNY, to open the door to understanding multimodal language used in expressing humor. The dataset and accompanying studies, present a framework in multimodal humor detection for the natural language processing community. UR-FUNNY is publicly available for research.

References in corpus (5)

• Adam: A Method for Stochastic Optimization
• MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videos
• Learning Factorized Multimodal Representations
• Efficient Low-rank Multimodal Fusion with Modality-Specific Factors
• Words Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviors

Cited by in corpus (12)

• Multi-modal Sarcasm Detection and Humor Classification in Code-mixed Conversations
• MIntRec: A New Dataset for Multimodal Intent Recognition
• MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis
• Multimodal Learning for Multi-Omics: A Survey
• MultiBench: Multiscale Benchmarks for Multimodal Representation Learning
• Hybrid Multimodal Fusion for Humor Detection
• ColBERT: Using BERT Sentence Embedding in Parallel Neural Networks for Computational Humor
• DeHumor: Visual Analytics for Decomposing Humor
• Uncertainty and Surprisal Jointly Deliver the Punchline: Exploiting Incongruity-Based Features for Humor Recognition
• TEASEL: A Transformer-Based Speech-Prefixed Language Model
• MELINDA: A Multimodal Dataset for Biomedical Experiment Method Classification
• Highlight Timestamp Detection Model for Comedy Videos via Multimodal Sentiment Analysis