29 papers
Towards Real-world Environment-aware Zero-shot Text-to-speech Synthesis via Disentangled Audio Infilling
Ye-Xin Lu, Xin Wang, Yang Ai +3
Recent zero-shot text-to-speech (TTS) systems achieve remarkable naturalness and speaker similarity but typically require high-quality speaker prompts and either strip away or enta…
CFMDCTCodec: A Low-Bitrate Neural Speech Codec with Noise-Prior-aware Conditional Flow Matching for MDCT-Spectral Enhancement
Xiao-Hang Jiang, Yang Ai, Hui-Peng Du +2
High-quality speech coding at low bitrates is crucial for bandwidth-constrained applications, yet remains challenging due to the severe loss of quality-critical information in high…
Ultra-Low-Bitrate Mel-Spectrogram-based Neural Speech Coding with Flow-Matching-based Refinement and Vocoding-driven Reconstruction
Hui-Peng Du, Yang Ai, Xiao-Hang Jiang +2
Ultra-low-bitrate speech coding is pivotal for bandwidth-constrained communication and deep compression, yet maintaining naturalness and speaker identity at such extreme bit budget…
LatentFlowSR: High-Fidelity Audio Super-Resolution via Noise-Robust Latent Flow Matching
Fei Liu, Yang Ai, Hui-Peng Du +2
Audio super-resolution aims to recover missing high-frequency details from bandwidth-limited low-resolution audio, thereby improving the naturalness and perceptual quality of the r…
Zero-Shot Knowledge Base Resizing for Rate-Adaptive Digital Semantic Communication
Shumin Yao, Hui Du, Lifeng Xie +4
Digital semantic communication systems, which often leverage the Vector Quantized Variational Autoencoder (VQ-VAE) framework, are pivotal for future wireless networks. In a VQ-VAE-…
CodeSep: Low-Bitrate Codec-Driven Speech Separation with Base-Token Disentanglement and Auxiliary-Token Serial Prediction
Hui-Peng Du, Yang Ai, Xiao-Hang Jiang +2
This paper targets a new scenario that integrates speech separation with speech compression, aiming to disentangle multiple speakers while producing discrete representations for ef…
Enhancing Noise Robustness for Neural Speech Codecs through Resource-Efficient Progressive Quantization Perturbation Simulation
Rui-Chen Zheng, Yang Ai, Hui-Peng Du +1
Noise robustness remains a critical challenge for deploying neural speech codecs in real-world acoustic scenarios where background noise is often inevitable. A key observation we m…
A Stage-Wise Learning Strategy with Fixed Anchors for Robust Speaker Verification
Bin Gu, Lipeng Dai, Huipeng Du +2
Learning robust speaker representations under noisy conditions presents significant challenges, which requires careful handling of both discriminative and noise-invariant propertie…
Universal Discrete-Domain Speech Enhancement
Fei Liu, Yang Ai, Ye-Xin Lu +3
In real-world scenarios, speech signals are inevitably corrupted by various types of interference, making speech enhancement (SE) a critical task for robust speech processing. Howe…
A High-Quality and Low-Complexity Streamable Neural Speech Codec with Knowledge Distillation
En-Wei Zhang, Hui-Peng Du, Xiao-Hang Jiang +2
While many current neural speech codecs achieve impressive reconstructed speech quality, they often neglect latency and complexity considerations, limiting their practical deployme…
A Distilled Low-Latency Neural Vocoder with Explicit Amplitude and Phase Prediction
Hui-Peng Du, Yang Ai, Zhen-Hua Ling
The majority of mainstream neural vocoders primarily focus on speech quality and generation speed, while overlooking latency, which is a critical factor in real-time applications.…
DAIEN-TTS: Disentangled Audio Infilling for Environment-Aware Text-to-Speech Synthesis
Ye-Xin Lu, Yu Gu, Kun Wei +3
This paper presents DAIEN-TTS, a zero-shot text-to-speech (TTS) framework that enables ENvironment-aware synthesis through Disentangled Audio Infilling. By leveraging separate spea…
Say More with Less: Variable-Frame-Rate Speech Tokenization via Adaptive Clustering and Implicit Duration Coding
Rui-Chen Zheng, Wenrui Liu, Hui-Peng Du +6
Existing speech tokenizers typically assign a fixed number of tokens per second, regardless of the varying information density or temporal fluctuations in the speech signal. This u…
Is GAN Necessary for Mel-Spectrogram-based Neural Vocoder?
Hui-Peng Du, Yang Ai, Rui-Chen Zheng +2
Recently, mainstream mel-spectrogram-based neural vocoders rely on generative adversarial network (GAN) for high-fidelity speech generation, e.g., HiFi-GAN and BigVGAN. However, th…
Vision-Integrated High-Quality Neural Speech Coding
Yao Guo, Yang Ai, Rui-Chen Zheng +3
This paper proposes a novel vision-integrated neural speech codec (VNSC), which aims to enhance speech coding quality by leveraging visual modality information. In VNSC, the image…
Improving Noise Robustness of LLM-based Zero-shot TTS via Discrete Acoustic Token Denoising
Ye-Xin Lu, Hui-Peng Du, Fei Liu +2
Large language model (LLM) based zero-shot text-to-speech (TTS) methods tend to preserve the acoustic environment of the audio prompt, leading to degradation in synthesized speech…
Pitch-and-Spectrum-Aware Singing Quality Assessment with Bias Correction and Model Fusion
Yu-Fei Shi, Yang Ai, Ye-Xin Lu +2
We participated in track 2 of the VoiceMOS Challenge 2024, which aimed to predict the mean opinion score (MOS) of singing samples. Our submission secured the first place among all…
Incremental Disentanglement for Environment-Aware Zero-Shot Text-to-Speech Synthesis
Ye-Xin Lu, Hui-Peng Du, Zheng-Yan Sheng +2
This paper proposes an Incremental Disentanglement-based Environment-Aware zero-shot text-to-speech (TTS) method, dubbed IDEA-TTS, that can synthesize speech for unseen speakers wh…
A Neural Denoising Vocoder for Clean Waveform Generation from Noisy Mel-Spectrogram based on Amplitude and Phase Predictions
Hui-Peng Du, Ye-Xin Lu, Yang Ai +1
This paper proposes a novel neural denoising vocoder that can generate clean speech waveforms from noisy mel-spectrograms. The proposed neural denoising vocoder consists of two com…
ESTVocoder: An Excitation-Spectral-Transformed Neural Vocoder Conditioned on Mel Spectrogram
Xiao-Hang Jiang, Hui-Peng Du, Yang Ai +2
This paper proposes ESTVocoder, a novel excitation-spectral-transformed neural vocoder within the framework of source-filter theory. The ESTVocoder transforms the amplitude and pha…
SAMOS: A Neural MOS Prediction Model Leveraging Semantic Representations and Acoustic Features
Yu-Fei Shi, Yang Ai, Ye-Xin Lu +2
Assessing the naturalness of speech using mean opinion score (MOS) prediction models has positive implications for the automatic evaluation of speech synthesis systems. Early MOS p…
MDCTCodec: A Lightweight MDCT-based Neural Audio Codec towards High Sampling Rate and Low Bitrate Scenarios
Xiao-Hang Jiang, Yang Ai, Rui-Chen Zheng +3
In this paper, we propose MDCTCodec, an efficient lightweight end-to-end neural audio codec based on the modified discrete cosine transform (MDCT). The encoder takes the MDCT spect…
APCodec+: A Spectrum-Coding-Based High-Fidelity and High-Compression-Rate Neural Audio Codec with Staged Training Paradigm
Hui-Peng Du, Yang Ai, Rui-Chen Zheng +1
This paper proposes a novel neural audio codec, named APCodec+, which is an improved version of APCodec. The APCodec+ takes the audio amplitude and phase spectra as the coding obje…
Stage-Wise and Prior-Aware Neural Speech Phase Prediction
Fei Liu, Yang Ai, Hui-Peng Du +3
This paper proposes a novel Stage-wise and Prior-aware Neural Speech Phase Prediction (SP-NSPP) model, which predicts the phase spectrum from input amplitude spectrum by two-stage…