88 citations · 305 across the 22 of their papers we have counts for
20 papers · 1 filter
Speech-DRAME: A Framework for Human-Aligned Benchmarks in Speech Role-Play
Jiatong Shi, Jionghao Han, Yichen Lu +6
Role-play has become a key testbed for generative models, expanding from text-only dialogue to multimodal interaction. Extending role-play to speech captures prosody, emotion, and…
Joint Semantic Knowledge Distillation and Masked Acoustic Modeling for Full-band Speech Restoration with Improved Intelligibility
Xiaoyu Liu, Xu Li, Joan Serrà +1
Speech restoration aims at restoring full-band speech with high quality and intelligibility, considering a diverse set of distortions. MaskSR is a recently proposed generative mode…
Masked Generative Video-to-Audio Transformers with Enhanced Synchronicity
Santiago Pascual, Chunghsin Yeh, Ioannis Tsiamas +1
Video-to-audio (V2A) generation leverages visual-only video features to render plausible sounds that match the scene. Importantly, the generated sound onsets should match the visua…
Sequential Contrastive Audio-Visual Learning
Ioannis Tsiamas, Santiago Pascual, Chunghsin Yeh +1
Contrastive learning has emerged as a powerful technique in audio-visual representation learning, leveraging the natural co-occurrence of audio and visual modalities in webscale vi…
GASS: Generalizing Audio Source Separation with Large-scale Data
Jordi Pons, Xiaoyu Liu, Santiago Pascual +1
Universal source separation targets at separating the audio sources of an arbitrary mix, removing the constraint to operate on a specific domain like speech or music. Yet, the pote…
CLIPSonic: Text-to-Audio Synthesis with Unlabeled Videos and Pretrained Language-Vision Models
Hao-Wen Dong, Xiaoyu Liu, Jordi Pons +5
Recent work has studied text-to-audio synthesis using large amounts of paired text-audio data. However, audio recordings with high-quality text annotations can be difficult to acqu…
Mono-to-stereo through parametric stereo generation
Joan Serrà, Davide Scaini, Santiago Pascual +4
Generating a stereophonic presentation from a monophonic audio signal is a challenging open task, especially if the goal is to obtain a realistic spatial imaging with a specific pa…
Full-band General Audio Synthesis with Score-based Diffusion
Santiago Pascual, Gautam Bhattacharya, Chunghsin Yeh +2
Recent works have shown the capability of deep generative models to tackle general audio synthesis from a single label, producing a variety of impulsive, tonal, and environmental s…
Adversarial Permutation Invariant Training for Universal Sound Separation
Emilian Postolache, Jordi Pons, Santiago Pascual +1
Universal sound separation consists of separating mixes with arbitrary sounds of different types, and permutation invariant training (PIT) is used to train source agnostic models t…
Universal Speech Enhancement with Score-based Diffusion
Joan Serrà, Santiago Pascual, Jordi Pons +2
Removing background noise from speech audio has been the subject of considerable effort, especially in recent years due to the rise of virtual communication and amateur recordings.…
On loss functions and evaluation metrics for music source separation
Enric Gusó, Jordi Pons, Santiago Pascual +1
We investigate which loss functions provide better separations via benchmarking an extensive set of those for music source separation. To that end, we first survey the most represe…
Upsampling layers for music source separation
Jordi Pons, Joan Serrà, Santiago Pascual +3
Upsampling artifacts are caused by problematic upsampling layers and due to spectral replicas that emerge while upsampling. Also, depending on the used upsampling layer, such artif…
Adversarial Auto-Encoding for Packet Loss Concealment
Santiago Pascual, Joan Serrà, Jordi Pons
Communication technologies like voice over IP operate under constrained real-time conditions, with voice packets being subject to delays and losses from the network. In such cases,…
Upsampling artifacts in neural audio synthesis
Jordi Pons, Santiago Pascual, Giulio Cengarle +1
A number of recent advances in neural audio synthesis rely on upsampling layers, which can introduce undesired artifacts. In computer vision, upsampling artifacts have been studied…
Sample Drop Detection for Distant-speech Recognition with Asynchronous Devices Distributed in Space
Tina Raissi, Santiago Pascual, Maurizio Omologo
In many applications of multi-microphone multi-device processing, the synchronization among different input channels can be affected by the lack of a common clock and isolated drop…
Problem-Agnostic Speech Embeddings for Multi-Speaker Text-to-Speech with SampleRNN
David Álvarez, Santiago Pascual, Antonio Bonafonte
Text-to-speech (TTS) acoustic models map linguistic features into an acoustic representation out of which an audible waveform is generated. The latest and most natural TTS systems…
Towards Generalized Speech Enhancement with Generative Adversarial Networks
Santiago Pascual, Joan Serrà, Antonio Bonafonte
The speech enhancement task usually consists of removing additive noise or reverberation that partially mask spoken utterances, affecting their intelligibility. However, little att…
Whispered-to-voiced Alaryngeal Speech Conversion with Generative Adversarial Networks
Santiago Pascual, Antonio Bonafonte, Joan Serrà +1
Most methods of voice restoration for patients suffering from aphonia either produce whispered or monotone speech. Apart from intelligibility, this type of speech lacks expressiven…
Self-Attention Linguistic-Acoustic Decoder
Santiago Pascual, Antonio Bonafonte, Joan Serrà
The conversion from text to speech relies on the accurate mapping from linguistic to acoustic symbol sequences, for which current practice employs recurrent statistical models like…
Language and Noise Transfer in Speech Enhancement Generative Adversarial Network
Santiago Pascual, Maruchan Park, Joan Serrà +2
Speech enhancement deep learning systems usually require large amounts of training data to operate in broad conditions or real applications. This makes the adaptability of those sy…