MM-ALT: A Multimodal Automatic Lyric Transcription System
arXiv:2207.06127 · doi:10.1145/3503161.3548411
Abstract
Automatic lyric transcription (ALT) is a nascent field of study attracting increasing interest from both the speech and music information retrieval communities, given its significant application potential. However, ALT with audio data alone is a notoriously difficult task due to instrumental accompaniment and musical constraints resulting in degradation of both the phonetic cues and the intelligibility of sung lyrics. To tackle this challenge, we propose the MultiModal Automatic Lyric Transcription system (MM-ALT), together with a new dataset, N20EM, which consists of audio recordings, videos of lip movements, and inertial measurement unit (IMU) data of an earbud worn by the performing singer. We first adapt the wav2vec 2.0 framework from automatic speech recognition (ASR) to the ALT task. We then propose a video-based ALT method and an IMU-based voice activity detection (VAD) method. In addition, we put forward the Residual Cross Attention (RCA) mechanism to fuse data from the three modalities (i.e., audio, video, and IMU). Experiments show the effectiveness of our proposed MM-ALT system, especially in terms of noise robustness. Project page is at https://n20em.github.io.
arXiv commentAccepted by ACM Multimedia 2022. Camera ready version and appendix
References in corpus (20)
- Adam: A Method for Stochastic Optimization
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Neural Machine Translation by Jointly Learning to Align and Translate
- Decoupled Weight Decay Regularization
- SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
- Gaussian Error Linear Units (GELUs)
- VoxCeleb2: Deep Speaker Recognition
- Attention-Based Models for Speech Recognition
- Deep Audio-Visual Speech Recognition
- SpeechBrain: A General-Purpose Speech Toolkit
- LRS3-TED: a large-scale dataset for visual speech recognition
- Attention Bottlenecks for Multimodal Fusion
- Perceiver IO: A General Architecture for Structured Inputs & Outputs
- Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection
- Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
- Recent Advances in End-to-End Automatic Speech Recognition
- Benchmarking Online Sequence-to-Sequence and Character-based Handwriting Recognition from IMU-Enhanced Pens
- As You Are, So Shall You Move Your Head: A System-Level Analysis between Head Movements and Corresponding Traits and Emotions
- HiP: Hierarchical Perceiver