257 citations · 424 across the 25 of their papers we have counts for
27 papers
Transcript-Supervised Post-Training of Generative Speech Enhancement on Real Recordings via Reinforce Adjoint Matching
Julius Richter, Christoph Boeddeker, Yoshiki Masuyama +4
We adapt Reinforce Adjoint Matching (RAM), a reward-based post-training method, to generative speech enhancement (SE). Starting from a pretrained SE model, RAM tilts the model's co…
Downstream-Task-Aware Unified Source Separation
Yoshiki Mitsui, Ryo Aihara, Tatsuhiko Saito +5
Task-aware unified source separation (TUSS) enables a single model to handle diverse separation tasks by conditioning on input prompts. However, conventional TUSS does not account…
Anomalous Sound Detection Meets Noise-Aware Self-Supervised Learning
Takuya Fujimura, Gordon Wichern, Yoshiki Masuyama +5
In this paper, we introduce noise-aware self-supervised learning (NA-SSL) models for noise-aware anomalous sound detection (NA-ASD). NA-ASD is an ASD task with two-channel audio re…
NABEATs: Noise-Aware Audio Representation Learning
Takuya Fujimura, Yoshiki Masuyama, Gordon Wichern +3
We propose the concept of noise-aware audio self-supervised learning (SSL), whose goal is to encode audio mixtures while suppressing undesired noise, and present Noise-Aware BEATs…
Technical Report for MERL's Real-TSE Challenge Submission
Dominik Klement, Yoshiki Masuyama, Christoph Boeddeker +4
Target speech extraction (TSE) has largely been dominated by neural network-based approaches trained and evaluated on synthetic fully overlapped data. The Real-TSE Challenge aims t…
Predictive-Generative Drift Decomposition for Speech Enhancement and Separation
Julius Richter, Yoshiki Masuyama, Christoph Boeddeker +3
We propose a plug-and-play framework for speech enhancement and separation that augments predictive methods with a generative speech prior. Our approach, termed Stochastic Interpol…
ArtiFree: Detecting and Reducing Generative Artifacts in Diffusion-based Speech Enhancement
Bhawana Chhaglani, Yang Gao, Julius Richter +4
Diffusion-based speech enhancement (SE) achieves natural-sounding speech and strong generalization, yet suffers from key limitations like generative artifacts and high inference la…
ReverbFX: A Dataset of Room Impulse Responses Derived from Reverb Effect Plugins for Singing Voice Dereverberation
Julius Richter, Till Svajda, Timo Gerkmann
We present ReverbFX, a new room impulse response (RIR) dataset designed for singing voice dereverberation research. Unlike existing datasets based on real recorded RIRs, ReverbFX f…
LipDiffuser: Lip-to-Speech Generation with Conditional Diffusion Models
Julius Richter, Danilo de Oliveira, Tal Peer +1
We present LipDiffuser, a conditional diffusion model for lip-to-speech generation synthesizing natural and intelligible speech directly from silent video recordings. Our approach…
Do We Need EMA for Diffusion-Based Speech Enhancement? Toward a Magnitude-Preserving Network Architecture
Julius Richter, Danilo de Oliveira, Timo Gerkmann
We study diffusion-based speech enhancement using a Schrodinger bridge formulation and extend the EDM2 framework to this setting. We employ time-dependent preconditioning of networ…
Non-intrusive Speech Quality Assessment with Diffusion Models Trained on Clean Speech
Danilo de Oliveira, Julius Richter, Jean-Marie Lemercier +2
Diffusion models have found great success in generating high quality, natural samples of speech, but their potential for density estimation for speech has so far remained largely u…
Investigating Training Objectives for Generative Speech Enhancement
Julius Richter, Danilo de Oliveira, Timo Gerkmann
Generative speech enhancement has recently shown promising advancements in improving speech quality in noisy environments. Multiple diffusion-based frameworks exist, each employing…
EARS: An Anechoic Fullband Speech Dataset Benchmarked for Speech Enhancement and Dereverberation
Julius Richter, Yi-Chiao Wu, Steven Krenn +5
We release the EARS (Expressive Anechoic Recordings of Speech) dataset, a high-quality speech dataset comprising 107 speakers from diverse backgrounds, totaling in 100 hours of cle…
The PESQetarian: On the Relevance of Goodhart's Law for Speech Enhancement
Danilo de Oliveira, Simon Welker, Julius Richter +1
To obtain improved speech enhancement models, researchers often focus on increasing performance according to specific instrumental metrics. However, when the same metric is used in…
Diffusion Models for Audio Restoration
Jean-Marie Lemercier, Julius Richter, Simon Welker +3
With the development of audio playback devices and fast data transmission, the demand for high sound quality is rising for both entertainment and communications. In this quest for…
Single and Few-step Diffusion for Generative Speech Enhancement
Bunlong Lay, Jean-Marie Lemercier, Julius Richter +1
Diffusion models have shown promising results in speech enhancement, using a task-adapted diffusion process for the conditional generation of clean speech given a noisy mixture. Ho…
On the Behavior of Intrusive and Non-intrusive Speech Enhancement Metrics in Predictive and Generative Settings
Danilo de Oliveira, Julius Richter, Jean-Marie Lemercier +2
Since its inception, the field of deep speech enhancement has been dominated by predictive (discriminative) approaches, such as spectral mapping or masking. Recently, however, nove…
Audio-Visual Speech Enhancement with Score-Based Generative Models
Julius Richter, Simone Frintrop, Timo Gerkmann
This paper introduces an audio-visual speech enhancement system that leverages score-based generative models, also known as diffusion models, conditioned on visual information. In…
Audio-Visual Speech Separation in Noisy Environments with a Lightweight Iterative Model
Héctor Martel, Julius Richter, Kai Li +2
We propose Audio-Visual Lightweight ITerative model (AVLIT), an effective and lightweight neural network that uses Progressive Learning (PL) to perform audio-visual speech separati…
Speech Signal Improvement Using Causal Generative Diffusion Models
Julius Richter, Simon Welker, Jean-Marie Lemercier +3
In this paper, we present a causal speech signal improvement system that is designed to handle different types of distortions. The method is based on a generative diffusion model w…
Reducing the Prior Mismatch of Stochastic Differential Equations for Diffusion-based Speech Enhancement
Bunlong Lay, Simon Welker, Julius Richter +1
Recently, score-based generative models have been successfully employed for the task of speech enhancement. A stochastic differential equation is used to model the iterative forwar…
StoRM: A Diffusion-based Stochastic Regeneration Model for Speech Enhancement and Dereverberation
Jean-Marie Lemercier, Julius Richter, Simon Welker +1
Diffusion models have shown a great ability at bridging the performance gap between predictive and generative approaches for speech enhancement. We have shown that they may even ou…
Analysing Diffusion-based Generative Approaches versus Discriminative Approaches for Speech Restoration
Jean-Marie Lemercier, Julius Richter, Simon Welker +1
Diffusion-based generative models have had a high impact on the computer vision and speech processing communities these past years. Besides data generation tasks, they have also be…
Speech Enhancement and Dereverberation with Diffusion-based Generative Models
Julius Richter, Simon Welker, Jean-Marie Lemercier +2
In this work, we build upon our previous publication and use diffusion-based generative models for speech enhancement. We present a detailed overview of the diffusion process that…