MM-ALT: A Multimodal Automatic Lyric Transcription System
arXiv:2207.06127 · doi:10.1145/3503161.3548411
Abstract
Automatic lyric transcription (ALT) is a nascent field of study attracting increasing interest from both the speech and music information retrieval communities, given its significant application potential. However, ALT with audio data alone is a notoriously difficult task due to instrumental accompaniment and musical constraints resulting in degradation of both the phonetic cues and the intelligibility of sung lyrics. To tackle this challenge, we propose the MultiModal Automatic Lyric Transcription system (MM-ALT), together with a new dataset, N20EM, which consists of audio recordings, videos of lip movements, and inertial measurement unit (IMU) data of an earbud worn by the performing singer. We first adapt the wav2vec 2.0 framework from automatic speech recognition (ASR) to the ALT task. We then propose a video-based ALT method and an IMU-based voice activity detection (VAD) method. In addition, we put forward the Residual Cross Attention (RCA) mechanism to fuse data from the three modalities (i.e., audio, video, and IMU). Experiments show the effectiveness of our proposed MM-ALT system, especially in terms of noise robustness. Project page is at https://n20em.github.io.
Accepted by ACM Multimedia 2022. Camera ready version and appendix
References in corpus (11)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Attention-Based Models for Speech Recognition
- SpeechBrain: A General-Purpose Speech Toolkit
- Attention Bottlenecks for Multimodal Fusion
- Perceiver IO: A General Architecture for Structured Inputs & Outputs
- Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection
- Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
- Recent Advances in End-to-End Automatic Speech Recognition
- Benchmarking Online Sequence-to-Sequence and Character-based Handwriting Recognition from IMU-Enhanced Pens
- As You Are, So Shall You Move Your Head: A System-Level Analysis between Head Movements and Corresponding Traits and Emotions
- HiP: Hierarchical Perceiver