VATLM: Visual-Audio-Text Pre-Training with Unified Masked Prediction for Speech Representation Learning
arXiv:2211.11275 · doi:10.1109/TMM.2023.3275873
Abstract
Although speech is a simple and effective way for humans to communicate with the outside world, a more realistic speech interaction contains multimodal information, e.g., vision, text. How to design a unified framework to integrate different modal information and leverage different resources (e.g., visual-audio pairs, audio-text pairs, unlabeled speech, and unlabeled text) to facilitate speech representation learning was not well explored. In this paper, we propose a unified cross-modal representation learning framework VATLM (Visual-Audio-Text Language Model). The proposed VATLM employs a unified backbone network to model the modality-independent information and utilizes three simple modality-dependent modules to preprocess visual, speech, and text inputs. In order to integrate these three modalities into one shared semantic space, VATLM is optimized with a masked prediction task of unified tokens, given by our proposed unified tokenizer. We evaluate the pre-trained VATLM on audio-visual related downstream tasks, including audio-visual speech recognition (AVSR), visual speech recognition (VSR) tasks. Results show that the proposed VATLM outperforms previous the state-of-the-art models, such as audio-visual pre-trained AV-HuBERT model, and analysis also demonstrates that VATLM is capable of aligning different modalities into the same space. To facilitate future research, we release the code and pre-trained models at https://aka.ms/vatlm.
arXiv comment11 pages, Accepted by IEEE Transactions on Multimedia
References in corpus (14)
- Representation Learning with Contrastive Predictive Coding
- VoxCeleb2: Deep Speaker Recognition
- WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
- Deep Audio-Visual Speech Recognition
- Self-Supervised Speech Representation Learning: A Review
- TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech
- LRS3-TED: a large-scale dataset for visual speech recognition
- TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptation
- GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audio
- Visual Speech Recognition for Multiple Languages in the Wild
- Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
- mSLAM: Massively multilingual joint pre-training for speech and text
- SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Training
- OPT: Omni-Perception Pre-Trainer for Cross-Modal Understanding and Generation