MPEcho: A Melody and Phoneme-Aware Generative Framework for Controllable Cover Song Generation
arXiv:2607.26698
The paper presents MPEcho, a generative framework for controllable cover song generation that incorporates both melody and explicit phoneme-level conditioning to improve lyric accuracy, and introduces Phonsa, a Whisper-based tool for high-precision phoneme transcription of singing voices.
Abstract
Cover song generation (CSG) should preserve the melodic and linguistic content of a reference song while recreating the remaining musical components. The state-of-the-art model SongEcho utilizes sequences and voiced/unvoiced (V/UV) tags for conditioning; however, implicit linguistic information from V/UV tags cannot guarantee lyric accuracy, leading to a high phoneme error rate (PER). Inspired by singing voice synthesis (SVS), we propose MPEcho, which integrates a phoneme encoder and a length regulator (LR) into the SongEcho framework. By providing explicit phoneme-level conditioning and precise temporal boundaries, MPEcho significantly reduces PER. To enable this, we developed Phonsa, a Whisper-based automatic transcription model that provides high-precision phoneme-level annotations for singing voices, overcoming the scarcity of high-quality audio-phoneme pairs. Experimental results validate the effectiveness of Phonsa for alignment and MPEcho for end-to-end CSG. The audio samples, code and weights can be accessed from https://lonian6.github.io/MPEcho.github.io/.
Accepted by the 27th International Society for Music Information Retrieval (ISMIR)