TEASEL: A Transformer-Based Speech-Prefixed Language Model

Abstract

Multimodal language analysis is a burgeoning field of NLP that aims to simultaneously model a speaker's words, acoustical annotations, and facial expressions. In this area, lexicon features usually outperform other modalities because they are pre-trained on large corpora via Transformer-based models. Despite their strong performance, training a new self-supervised learning (SSL) Transformer on any modality is not usually attainable due to insufficient data, which is the case in multimodal language learning. This work proposes a Transformer-Based Speech-Prefixed Language Model called TEASEL to approach the mentioned constraints without training a complete Transformer model. TEASEL model includes speech modality as a dynamic prefix besides the textual modality compared to a conventional language model. This method exploits a conventional pre-trained language model as a cross-modal Transformer model. We evaluated TEASEL for the multimodal sentiment analysis task defined by CMU-MOSI dataset. Extensive experiments show that our model outperforms unimodal baseline language models by 4% and outperforms the current multimodal state-of-the-art (SoTA) model by 1% in F1-score. Additionally, our proposed method is 72% smaller than the SoTA model.

References in corpus (14)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Gaussian Error Linear Units (GELUs)
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• Language Models are Few-Shot Learners
• VoxCeleb2: Deep Speaker Recognition
• MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videos
• Learning Factorized Multimodal Representations
• UR-FUNNY: A Multimodal Language Dataset for Understanding Humor
• Memory Fusion Network for Multi-view Sequential Learning
• Pretrained Transformers as Universal Computation Engines
• Efficient Low-rank Multimodal Fusion with Modality-Specific Factors
• Multimodal Few-Shot Learning with Frozen Language Models
• Factorized Multimodal Transformer for Multimodal Sequential Learning