19 citations · 49 across the 31 of their papers we have counts for
32 papers
Controlling Backchannels in Streamable Full-duplex Models
Maike Züfle, Peter Polák, Sefik Emre Eskimez +3
Backchannels, brief acknowledgements like "uh-huh" produced while the other party may still be talking, are central to natural conversation, but full-duplex spoken dialogue models…
TurnBench: A Multi-Domain Benchmark for Turn-Taking Dynamics in Spoken Dialogue
Freeman Jiang, Ramon Sanabria, Soham Deshmukh +16
Speakers in natural conversation take turns speaking and listening, deciding in real time when to take, hold, or yield the floor. However, turn-taking evaluation remains limited du…
Neural Speech Extraction with Human Feedback
Malek Itani, Ashton Graves, Sefik Emre Eskimez +1
We present the first neural target speech extraction (TSE) system that uses human feedback for iterative refinement. Our approach allows users to mark specific segments of the TSE…
TS3-Codec: Transformer-Based Simple Streaming Single Codec
Haibin Wu, Naoyuki Kanda, Sefik Emre Eskimez +1
Neural audio codecs (NACs) have garnered significant attention as key technologies for audio compression as well as audio representation for speech language models. While mainstrea…
Target conversation extraction: Source separation using turn-taking dynamics
Tuochao Chen, Qirui Wang, Bohan Wu +4
Extracting the speech of participants in a conversation amidst interfering speakers and noise presents a challenging problem. In this paper, we introduce the novel task of target c…
Knowledge boosting during low-latency inference
Vidya Srinivas, Malek Itani, Tuochao Chen +3
Models for low-latency, streaming applications could benefit from the knowledge capacity of larger models, but edge devices cannot run these models due to resource constraints. A p…
Laugh Now Cry Later: Controlling Time-Varying Emotional States of Flow-Matching-Based Zero-Shot Text-to-Speech
Haibin Wu, Xiaofei Wang, Sefik Emre Eskimez +8
People change their tones of voice, often accompanied by nonverbal vocalizations (NVs) such as laughter and cries, to convey rich emotions. However, most text-to-speech (TTS) syste…
An Investigation of Noise Robustness for Flow-Matching-Based Zero-Shot TTS
Xiaofei Wang, Sefik Emre Eskimez, Manthan Thakker +8
Recently, zero-shot text-to-speech (TTS) systems, capable of synthesizing any speaker's voice from a short audio prompt, have made rapid advancements. However, the quality of the g…
Total-Duration-Aware Duration Modeling for Text-to-Speech Systems
Sefik Emre Eskimez, Xiaofei Wang, Manthan Thakker +9
Accurate control of the total duration of generated speech by adjusting the speech rate is crucial for various text-to-speech (TTS) applications. However, the impact of adjusting t…
E2 TTS: Embarrassingly Easy Fully Non-Autoregressive Zero-Shot TTS
Sefik Emre Eskimez, Xiaofei Wang, Manthan Thakker +10
This paper introduces Embarrassingly Easy Text-to-Speech (E2 TTS), a fully non-autoregressive zero-shot text-to-speech system that offers human-level naturalness and state-of-the-a…
Making Flow-Matching-Based Zero-Shot Text-to-Speech Laugh as You Like
Naoyuki Kanda, Xiaofei Wang, Sefik Emre Eskimez +12
Laughter is one of the most expressive and natural aspects of human speech, conveying emotions, social cues, and humor. However, most text-to-speech (TTS) systems lack the ability…
SpeechX: Neural Codec Language Model as a Versatile Speech Transformer
Xiaofei Wang, Manthan Thakker, Zhuo Chen +7
Recent advancements in generative speech models based on audio-text prompts have enabled remarkable innovations like high-quality zero-shot text-to-speech. However, existing models…
Real-Time Audio-Visual End-to-End Speech Enhancement
Zirun Zhu, Hemin Yang, Min Tang +3
Audio-visual speech enhancement (AV-SE) methods utilize auxiliary visual cues to enhance speakers' voices. Therefore, technically they should be able to outperform the audio-only s…
ICASSP 2023 Deep Noise Suppression Challenge
Harishchandra Dubey, Ashkan Aazami, Vishak Gopal +9
Deep Speech Enhancement Challenge is the 5th edition of deep noise suppression (DNS) challenges organized at ICASSP 2023 Signal Processing Grand Challenges. DNS challenges were org…
Speech separation with large-scale self-supervised learning
Zhuo Chen, Naoyuki Kanda, Jian Wu +6
Self-supervised learning (SSL) methods such as WavLM have shown promising speech separation (SS) results in small-scale simulation-based experiments. In this work, we extend the ex…
Breaking the trade-off in personalized speech enhancement with cross-task knowledge distillation
Hassan Taherian, Sefik Emre Eskimez, Takuya Yoshioka
Personalized speech enhancement (PSE) models achieve promising results compared with unconditional speech enhancement models due to their ability to remove interfering speech in ad…
Real-Time Joint Personalized Speech Enhancement and Acoustic Echo Cancellation
Sefik Emre Eskimez, Takuya Yoshioka, Alex Ju +3
Personalized speech enhancement (PSE) is a real-time SE approach utilizing a speaker embedding of a target person to remove background noise, reverberation, and interfering voices.…
Leveraging Real Conversational Data for Multi-Channel Continuous Speech Separation
Xiaofei Wang, Dongmei Wang, Naoyuki Kanda +2
Existing multi-channel continuous speech separation (CSS) models are heavily dependent on supervised data - either simulated data which causes data mismatch between the training an…
Fast Real-time Personalized Speech Enhancement: End-to-End Enhancement Network (E3Net) and Knowledge Distillation
Manthan Thakker, Sefik Emre Eskimez, Takuya Yoshioka +1
This paper investigates how to improve the runtime speed of personalized speech enhancement (PSE) networks while maintaining the model quality. Our approach includes two aspects: a…
Sequence-level self-learning with multiple hypotheses
Kenichi Kumatani, Dimitrios Dimitriadis, Yashesh Gaur +4
In this work, we develop new self-learning techniques with an attention-based sequence-to-sequence (seq2seq) model for automatic speech recognition (ASR). For untranscribed speech…
Separating Long-Form Speech with Group-Wise Permutation Invariant Training
Wangyou Zhang, Zhuo Chen, Naoyuki Kanda +8
Multi-talker conversational speech processing has drawn many interests for various applications such as meeting transcription. Speech separation is often required to handle overlap…
One model to enhance them all: array geometry agnostic multi-channel personalized speech enhancement
Hassan Taherian, Sefik Emre Eskimez, Takuya Yoshioka +3
With the recent surge of video conferencing tools usage, providing high-quality speech signals and accurate captions have become essential to conduct day-to-day business or connect…
Personalized Speech Enhancement: New Models and Comprehensive Evaluation
Sefik Emre Eskimez, Takuya Yoshioka, Huaming Wang +3
Personalized speech enhancement (PSE) models utilize additional cues, such as speaker embeddings like d-vectors, to remove background noise and interfering speech in real-time and…
All-neural beamformer for continuous speech separation
Zhuohuang Zhang, Takuya Yoshioka, Naoyuki Kanda +4
Continuous speech separation (CSS) aims to separate overlapping voices from a continuous influx of conversational audio containing an unknown number of utterances spoken by an unkn…