111 citations · 267 across the 29 of their papers we have counts for
29 papers
Stable Autoregressive Speech Generation with Low-Frame-Rate High-Dimensional Continuous Tokens
Yi Luo, Rongzhi Gu, Jixun Yao
Balancing sequence length, representational capacity, and long-horizon stability is a central problem in autoregressive (AR) speech and audio generation. Representations with highe…
SongPrep: A Preprocessing Framework and End-to-end Model for Full-song Structure Parsing and Lyrics Transcription
Wei Tan, Shun Lei, Huaicheng Zhang +6
Artificial Intelligence Generated Content (AIGC) is currently a popular research area. Among its various branches, song generation has attracted growing interest. Despite the abund…
WAKE: Watermarking Audio with Key Enrichment
Yaoxun Xu, Jianwei Yu, Hangting Chen +5
As deep learning advances in audio generation, challenges in audio security and copyright protection highlight the need for robust audio watermarking. Recent neural network-based m…
SongEditor: Adapting Zero-Shot Song Generation Language Model as a Multi-Task Editor
Chenyu Yang, Shuai Wang, Hangting Chen +7
The emergence of novel generative modeling paradigms, particularly audio language models, has significantly advanced the field of song generation. Although state-of-the-art models…
MuQ: Self-Supervised Music Representation Learning with Mel Residual Vector Quantization
Haina Zhu, Yizhi Zhou, Hangting Chen +6
Recent years have witnessed the success of foundation models pre-trained with self-supervised learning (SSL) in various music informatics understanding tasks, including music taggi…
MuCodec: Ultra Low-Bitrate Music Codec
Yaoxun Xu, Hangting Chen, Jianwei Yu +5
Music codecs are a vital aspect of audio codec research, and ultra low-bitrate compression holds significant importance for music transmission and generation. Due to the complexity…
Gull: A Generative Multifunctional Audio Codec
Yi Luo, Jianwei Yu, Hangting Chen +2
We introduce Gull, a generative multifunctional audio codec. Gull is a general purpose neural audio compression and decompression model which can be applied to a wide range of task…
SECap: Speech Emotion Captioning with Large Language Model
Yaoxun Xu, Hangting Chen, Jianwei Yu +6
Speech emotions are crucial in human communication and are extensively used in fields like speech synthesis and natural language understanding. Most prior studies, such as speech e…
ReZero: Region-customizable Sound Extraction
Rongzhi Gu, Yi Luo
We introduce region-customizable sound extraction (ReZero), a general and flexible framework for the multi-channel region-wise sound extraction (R-SE) task. R-SE task aims at extra…
Ultra Dual-Path Compression For Joint Echo Cancellation And Noise Suppression
Hangting Chen, Jianwei Yu, Yi Luo +4
Echo cancellation and noise reduction are essential for full-duplex communication, yet most existing neural networks have high computational costs and are inflexible in tuning mode…
The Sound Demixing Challenge 2023 $\unicode{x2013}$ Cinematic Demixing Track
Stefan Uhlich, Giorgio Fabbro, Masato Hirano +14
This paper summarizes the cinematic demixing (CDX) track of the Sound Demixing Challenge 2023 (SDX'23). We provide a comprehensive summary of the challenge setup, detailing the str…
Fast Random Approximation of Multi-channel Room Impulse Response
Yi Luo, Rongzhi Gu
Modern neural-network-based speech processing systems are typically required to be robust against reverberation, and the training of such systems thus needs a large amount of rever…
3D Neural Beamforming for Multi-channel Speech Separation Against Location Uncertainty
Rongzhi Gu, Shi-Xiong Zhang, Dong Yu
Multi-channel speech separation using speaker's directional information has demonstrated significant gains over blind speech separation. However, it has two limitations. First, sub…
Towards Unified All-Neural Beamforming for Time and Frequency Domain Speech Separation
Rongzhi Gu, Shi-Xiong Zhang, Yuexian Zou +1
Recently, frequency domain all-neural beamforming methods have achieved remarkable progress for multichannel speech separation. In parallel, the integration of time domain network…
Probing Deep Speaker Embeddings for Speaker-related Tasks
Zifeng Zhao, Ding Pan, Junyi Peng +1
Deep speaker embeddings have shown promising results in speaker recognition, as well as in other speaker-related tasks. However, some issues are still under explored, for instance,…
High Fidelity Speech Enhancement with Band-split RNN
Jianwei Yu, Yi Luo, Hangting Chen +2
Despite the rapid progress in speech enhancement (SE) research, enhancing the quality of desired speech in environments with strong noise and interfering speakers remains challengi…
Parameter-efficient transfer learning of pre-trained Transformer models for speaker verification using adapters
Junyi Peng, Themos Stafylakis, Rongzhi Gu +4
Recently, the pre-trained Transformer models have received a rising interest in the field of speech processing thanks to their great success in various downstream tasks. However, m…
Improving Dual-Microphone Speech Enhancement by Learning Cross-Channel Features with Multi-Head Attention
Xinmeng Xu, Rongzhi Gu, Yuexian Zou
Hand-crafted spatial features, such as inter-channel intensity difference (IID) and inter-channel phase difference (IPD), play a fundamental role in recent deep learning based dual…
Speaker-Aware Mixture of Mixtures Training for Weakly Supervised Speaker Extraction
Zifeng Zhao, Rongzhi Gu, Dongchao Yang +2
Dominant researches adopt supervised training for speaker extraction, while the scarcity of ideally clean corpus and channel mismatch problem are rarely considered. To this end, we…
Target Confusion in End-to-end Speaker Extraction: Analysis and Approaches
Zifeng Zhao, Dongchao Yang, Rongzhi Gu +2
Recently, end-to-end speaker extraction has attracted increasing attention and shown promising results. However, its performance is often inferior to that of a blind source separat…
Learning Decoupling Features Through Orthogonality Regularization
Li Wang, Rongzhi Gu, Weiji Zhuang +3
Keyword spotting (KWS) and speaker verification (SV) are two important tasks in speech applications. Research shows that the state-of-art KWS and SV models are trained independentl…
Text Anchor Based Metric Learning for Small-footprint Keyword Spotting
Li Wang, Rongzhi Gu, Nuo Chen +1
Keyword Spotting (KWS) remains challenging to achieve the trade-off between small footprint and high accuracy. Recently proposed metric learning approaches improved the generalizab…
Complex Neural Spatial Filter: Enhancing Multi-channel Target Speech Separation in Complex Domain
Rongzhi Gu, Shi-Xiong Zhang, Yuexian Zou +1
To date, mainstream target speech separation (TSS) approaches are formulated to estimate the complex ratio mask (cRM) of the target speech in time-frequency domain under supervised…
Layer Reduction: Accelerating Conformer-Based Self-Supervised Model via Layer Consistency
Jinchuan Tian, Rongzhi Gu, Helin Wang +1
Transformer-based self-supervised models are trained as feature extractors and have empowered many downstream speech tasks to achieve state-of-the-art performance. However, both th…