Timo Gerkmann

Perceptual Quality Loss or Loss of Perceptual Quality?

Danilo de Oliveira, Tal Peer, Maurício do V. M. da Costa +1

Contemporary deep speech enhancement (SE) models are often trained with specific auxiliary terms in the loss function as a way to improve their performance in terms of perceptual m…

Directivity-Conditioned Low-Latency Neural Filtering for Speech Enhancement in Hearing Aids

Lennart Uphaus, André Merboldt, Markus Hofbauer +1

Latest advances in neural directional filtering show exceptional results in adapting the direction and shape of directivity patterns during the inference phase. However, in the exi…

EffVOC: Low-Delay Efficient Speech Waveform Reconstruction from Spectral Representations Without Phase

Renzheng Shi, Simon Welker, Timo Gerkmann +1

The Griffin-Lim algorithm has been a seminal contribution for phase reconstruction from amplitude spectrograms, however, requiring (infinitely) high algorithmic delay. Its low-dela…

Repurposing a Speech Classifier for Guided Diffusion-Based Speech Generation

Rostislav Makarov, Timo Gerkmann

Classifier guidance is a way to control diffusion generation by using a noise-conditioned classifier to steer the sampling process toward a target class. One drawback of classifier…

Your U-Net Dereverberation Model is Secretly an RIR Encoder

Sina Khanagha, Timo Gerkmann

In this work, we analyze the ability of NCSN++ U-Net based audio dereverberation models to capture global room characteristics in their intermediate representations. Through an emp…

Too Good to Be True: A Study on Modern Automatic Speech Recognition for the Evaluation of Speech Enhancement

Danilo de Oliveira, Tal Peer, Timo Gerkmann

Speech enhancement (SE) systems are typically evaluated using a variety of instrumental metrics. The use of automatic speech recognition (ASR) systems to evaluate SE performance is…

A Fast Solver for Interpolating Stochastic Differential Equation Diffusion Models for Speech Restoration

Bunlong Lay, Timo Gerkmann

Diffusion Probabilistic Models (DPMs) are a well-established class of diffusion models for unconditional image generation, while SGMSE+ is a well-established conditional diffusion…

Bone-conduction Guided Multimodal Speech Enhancement with Conditional Diffusion Models

Sina Khanagha, Bunlong Lay, Timo Gerkmann

Single-channel speech enhancement models face significant performance degradation in extremely noisy environments. While prior work has shown that complementary bone-conducted spee…

Are These Even Words? Quantifying the Gibberishness of Generative Speech Models

Danilo de Oliveira, Tal Peer, Jonas Rochdi +1

Significant research efforts are currently being dedicated to non-intrusive quality and intelligibility assessment, especially given how it enables curation of large scale datasets…

Diffusion Buffer for Online Generative Speech Enhancement

Bunlong Lay, Rostislav Makarov, Simon Welker +2

Online Speech Enhancement was mainly reserved for predictive models. A key advantage of these models is that for an incoming signal frame from a stream of data, the model is called…

Real-Time Streaming Mel Vocoding with Generative Flow Matching

Simon Welker, Tal Peer, Timo Gerkmann

The task of Mel vocoding, i.e., the inversion of a Mel magnitude spectrogram to an audio waveform, is still a key component in many text-to-speech (TTS) systems today. Based on gen…

Are Modern Speech Enhancement Systems Vulnerable to Adversarial Attacks?

Rostislav Makarov, Lea Schönherr, Timo Gerkmann

Machine learning approaches for speech enhancement are becoming increasingly expressive, enabling ever more powerful modifications of input signals. In this paper, we demonstrate t…

Enhancing In-the-Wild Speech Emotion Conversion with Resynthesis-based Duration Modeling

Navin Raj Prabhu, Danilo de Oliveira, Nale Lehmann-Willenbrock +1

Speech Emotion Conversion aims to modify the emotion expressed in input speech while preserving lexical content and speaker identity. Recently, generative modeling approaches have…

Diffusion Buffer: Online Diffusion-based Speech Enhancement with Sub-Second Latency

Bunlong Lay, Rostislav Makarov, Timo Gerkmann

Diffusion models are a class of generative models that have been recently used for speech enhancement with remarkable success but are computationally expensive at inference time. T…

ReverbFX: A Dataset of Room Impulse Responses Derived from Reverb Effect Plugins for Singing Voice Dereverberation

Julius Richter, Till Svajda, Timo Gerkmann

We present ReverbFX, a new room impulse response (RIR) dataset designed for singing voice dereverberation research. Unlike existing datasets based on real recorded RIRs, ReverbFX f…

LipDiffuser: Lip-to-Speech Generation with Conditional Diffusion Models

Julius Richter, Danilo de Oliveira, Tal Peer +1

We present LipDiffuser, a conditional diffusion model for lip-to-speech generation synthesizing natural and intelligible speech directly from silent video recordings. Our approach…

Do We Need EMA for Diffusion-Based Speech Enhancement? Toward a Magnitude-Preserving Network Architecture

Julius Richter, Danilo de Oliveira, Timo Gerkmann

We study diffusion-based speech enhancement using a Schrodinger bridge formulation and extend the EDM2 framework to this setting. We employ time-dependent preconditioning of networ…

HRTF Estimation using a Score-based Prior

Etienne Thuillier, Jean-Marie Lemercier, Eloi Moliner +2

We present a head-related transfer function (HRTF) estimation method which relies on a data-driven prior given by a score-based diffusion model. The HRTF is estimated in reverberan…

Non-intrusive Speech Quality Assessment with Diffusion Models Trained on Clean Speech

Danilo de Oliveira, Julius Richter, Jean-Marie Lemercier +2

Diffusion models have found great success in generating high quality, natural samples of speech, but their potential for density estimation for speech has so far remained largely u…

Investigating Training Objectives for Generative Speech Enhancement

Julius Richter, Danilo de Oliveira, Timo Gerkmann

Generative speech enhancement has recently shown promising advancements in improving speech quality in noisy environments. Multiple diffusion-based frameworks exist, each employing…

Unsupervised Blind Joint Dereverberation and Room Acoustics Estimation with Diffusion Models

Jean-Marie Lemercier, Eloi Moliner, Simon Welker +2

This paper presents an unsupervised method for single-channel blind dereverberation and room impulse response (RIR) estimation, called BUDDy. The algorithm is rooted in Bayesian po…

Robustness of Speech Separation Models for Similar-pitch Speakers

Bunlong Lay, Sebastian Zaczek, Kristina Tesch +1

Single-channel speech separation is a crucial task for enhancing speech recognition systems in multi-speaker environments. This paper investigates the robustness of state-of-the-ar…

EARS: An Anechoic Fullband Speech Dataset Benchmarked for Speech Enhancement and Dereverberation

Julius Richter, Yi-Chiao Wu, Steven Krenn +5

We release the EARS (Expressive Anechoic Recordings of Speech) dataset, a high-quality speech dataset comprising 107 speakers from diverse backgrounds, totaling in 100 hours of cle…