542 citations · 1.5k across the 44 of their papers we have counts for
44 papers
AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining
Haohe Liu, Yi Yuan, Xubo Liu +7
Although audio generation shares commonalities across different types of audio, such as speech, music, and sound effects, designing models for each type requires careful considerat…
Separate Anything You Describe
Xubo Liu, Qiuqiang Kong, Yan Zhao +7
Language-queried audio source separation (LASS) is a new paradigm for computational auditory scene analysis (CASA). LASS aims to separate a target sound from an audio mixture given…
PolyVoice: Language Models for Speech to Speech Translation
Qianqian Dong, Zhiying Huang, Qiao Tian +15
We propose PolyVoice, a language model-based framework for speech-to-speech translation (S2ST) system. Our framework consists of two language models: a translation language model a…
Efficient Neural Music Generation
Max W. Y. Lam, Qiao Tian, Tang Li +10
Recent progress in music generation has been remarkably advanced by the state-of-the-art MusicLM, which comprises a hierarchy of three LMs, respectively, for semantic, coarse acous…
Language-universal phonetic encoder for low-resource speech recognition
Siyuan Feng, Ming Tu, Rui Xia +2
Multilingual training is effective in improving low-resource ASR, which may partially be explained by phonetic representation sharing between languages. In end-to-end (E2E) ASR sys…
Language-Universal Phonetic Representation in Multilingual Speech Pretraining for Low-Resource Speech Recognition
Siyuan Feng, Ming Tu, Rui Xia +2
We improve low-resource ASR by integrating the ideas of multilingual training and self-supervised learning. Concretely, we leverage an International Phonetic Alphabet (IPA) multili…
A unified front-end framework for English text-to-speech synthesis
Zelin Ying, Chen Li, Yu Dong +4
The front-end is a critical component of English text-to-speech (TTS) systems, responsible for extracting linguistic features that are essential for a text-to-speech model to synth…
Joint Multi-scale Cross-lingual Speaking Style Transfer with Bidirectional Attention Mechanism for Automatic Dubbing
Jingbei Li, Sipan Li, Ping Chen +7
Automatic dubbing, which generates a corresponding version of the input speech in another language, could be widely utilized in many real-world scenarios such as video and game loc…
Memory Augmented Lookup Dictionary based Language Modeling for Automatic Speech Recognition
Yukun Feng, Ming Tu, Rui Xia +2
Recent studies have shown that using an external Language Model (LM) benefits the end-to-end Automatic Speech Recognition (ASR). However, predicting tokens that appear less frequen…
Streaming Voice Conversion Via Intermediate Bottleneck Features And Non-streaming Teacher Guidance
Yuanzhe Chen, Ming Tu, Tang Li +7
Streaming voice conversion (VC) is the task of converting the voice of one person to another in real-time. Previous streaming VC methods use phonetic posteriorgrams (PPGs) extracte…
Neural Sound Field Decomposition with Super-resolution of Sound Direction
Qiuqiang Kong, Shilei Liu, Junjie Shi +5
Sound field decomposition predicts waveforms in arbitrary directions using signals from a limited number of microphones as inputs. Sound field decomposition is fundamental to downs…
Cloning one's voice using very limited data in the wild
Dongyang Dai, Yuanzhe Chen, Li Chen +6
With the increasing popularity of speech synthesis products, the industry has put forward more requirements for personalized speech synthesis: (1) How to use low-resource, easily a…
VoiceFixer: Toward General Speech Restoration with Neural Vocoder
Haohe Liu, Qiuqiang Kong, Qiao Tian +4
Speech restoration aims to remove distortions in speech signals. Prior methods mainly focus on single-task speech restoration (SSR), such as speech denoising or speech declipping.…
Decoupling Magnitude and Phase Estimation with Deep ResUNet for Music Source Separation
Qiuqiang Kong, Yin Cao, Haohe Liu +2
Deep neural network based methods have been successfully applied to music source separation. They typically learn a mapping from a mixture spectrogram to a set of source spectrogra…
The ByteDance Speaker Diarization System for the VoxCeleb Speaker Recognition Challenge 2021
Keke Wang, Xudong Mao, Hao Wu +4
This paper describes the ByteDance speaker diarization system for the fourth track of the VoxCeleb Speaker Recognition Challenge 2021 (VoxSRC-21). The VoxSRC-21 provides both the d…
Joint Echo Cancellation and Noise Suppression based on Cascaded Magnitude and Complex Mask Estimation
Xiaofeng Shu, Yehang Zhu, Yanjie Chen +4
Acoustic echo and background noise can seriously degrade the intelligibility of speech. In practice, echo and noise suppression are usually treated as two separated tasks and can b…
Audiovisual Singing Voice Separation
Bochen Li, Yuxuan Wang, Zhiyao Duan
Separating a song into vocal and accompaniment components is an active research topic, and recent years witnessed an increased performance from supervised training using deep learn…
Supervised Chorus Detection for Popular Music Using Convolutional Neural Network and Multi-task Learning
Ju-Chiang Wang, Jordan B. L. Smith, Jitong Chen +2
This paper presents a novel supervised approach to detecting the chorus segments in popular music. Traditional approaches to this task are mostly unsupervised, with pipelines desig…
Modeling the Compatibility of Stem Tracks to Generate Music Mashups
Jiawen Huang, Ju-Chiang Wang, Jordan B. L. Smith +2
A music mashup combines audio elements from two or more songs to create a new work. To reduce the time and effort required to make them, researchers have developed algorithms that…
Listen, Read, and Identify: Multimodal Singing Language Identification of Music
Keunwoo Choi, Yuxuan Wang
We propose a multimodal singing language classification model that uses both audio content and textual metadata. LRID-Net, the proposed model, takes an audio signal and a language…
Speech enhancement with weakly labelled data from AudioSet
Qiuqiang Kong, Haohe Liu, Xingjian Du +3
Speech enhancement is a task to improve the intelligibility and perceptual quality of degraded speech signal. Recently, neural networks based methods have been applied to speech en…
CatNet: music source separation system with mix-audio augmentation
Xuchen Song, Qiuqiang Kong, Xingjian Du +1
Music source separation (MSS) is the task of separating a music piece into individual sources, such as vocals and accompaniment. Recently, neural network based methods have been ap…
Large-Scale MIDI-based Composer Classification
Qiuqiang Kong, Keunwoo Choi, Yuxuan Wang
Music classification is a task to classify a music piece into labels such as genres or composers. We propose large-scale MIDI based composer classification systems using GiantMIDI-…
GiantMIDI-Piano: A large-scale MIDI dataset for classical piano music
Qiuqiang Kong, Bochen Li, Jitong Chen +1
Symbolic music datasets are important for music information retrieval and musical analysis. However, there is a lack of large-scale symbolic datasets for classical piano music. In…