788 citations · 2.9k across the 245 of their papers we have counts for
41 papers · 1 filter
Speech collage: code-switched audio generation by collaging monolingual corpora
Amir Hussein, Dorsa Zeinali, Ondřej Klejch +6
Designing effective automatic speech recognition (ASR) systems for Code-Switching (CS) often depends on the availability of the transcribed CS resources. To address data scarcity,…
Improving Audio Captioning Models with Fine-grained Audio Features, Text Embedding Supervision, and LLM Mix-up Augmentation
Shih-Lun Wu, Xuankai Chang, Gordon Wichern +4
Automated audio captioning (AAC) aims to generate informative descriptions for various sounds from nature and/or human activities. In recent years, AAC has quickly attracted resear…
Exploring the Integration of Speech Separation and Recognition with Self-Supervised Learning Representation
Yoshiki Masuyama, Xuankai Chang, Wangyou Zhang +5
Neural speech separation has made remarkable progress and its integration with automatic speech recognition (ASR) is an important direction towards realizing multi-speaker ASR. Thi…
Exploration on HuBERT with Multiple Resolutions
Jiatong Shi, Yun Tang, Hirofumi Inaguma +3
Hidden-unit BERT (HuBERT) is a widely-used self-supervised learning (SSL) model in speech processing. However, we argue that its fixed 20ms resolution for hidden representations wo…
UNSSOR: Unsupervised Neural Speech Separation by Leveraging Over-determined Training Mixtures
Zhong-Qiu Wang, Shinji Watanabe
In reverberant conditions with multiple concurrent speakers, each microphone acquires a mixture signal of multiple speakers at a different location. In over-determined conditions w…
ML-SUPERB: Multilingual Speech Universal PERformance Benchmark
Jiatong Shi, Dan Berrebbi, William Chen +8
Speech processing Universal PERformance Benchmark (SUPERB) is a leaderboard to benchmark the performance of Self-Supervised Learning (SSL) models on various speech processing tasks…
Enhancing Speech-to-Speech Translation with Multiple TTS Targets
Jiatong Shi, Yun Tang, Ann Lee +4
It has been known that direct speech-to-speech translation (S2ST) models usually suffer from the data scarcity issue because of the limited existing parallel materials for both sou…
ESPnet-ST-v2: Multipurpose Spoken Language Translation Toolkit
Brian Yan, Jiatong Shi, Yun Tang +13
ESPnet-ST-v2 is a revamp of the open-source ESPnet-ST toolkit necessitated by the broadening interests of the spoken language translation community. ESPnet-ST-v2 supports 1) offlin…
PAAPLoss: A Phonetic-Aligned Acoustic Parameter Loss for Speech Enhancement
Muqiao Yang, Joseph Konan, David Bick +5
Despite rapid advancement in recent years, current speech enhancement models often produce speech that differs in perceptual quality from real clean speech. We propose a learning o…
4D ASR: Joint modeling of CTC, Attention, Transducer, and Mask-Predict decoders
Yui Sudo, Muhammad Shakeel, Brian Yan +2
The network architecture of end-to-end (E2E) automatic speech recognition (ASR) can be classified into several models, including connectionist temporal classification (CTC), recurr…
TF-GridNet: Integrating Full- and Sub-Band Modeling for Speech Separation
Zhong-Qiu Wang, Samuele Cornell, Shukjae Choi +3
We propose TF-GridNet for speech separation. The model is a novel deep neural network (DNN) integrating full- and sub-band modeling in the time-frequency (T-F) domain. It stacks se…
Large-scale learning of generalised representations for speaker recognition
Jee-weon Jung, Hee-Soo Heo, Bong-Jin Lee +5
The objective of this work is to develop a speaker recognition model to be used in diverse scenarios. We hypothesise that two components should be adequately configured to build su…
In search of strong embedding extractors for speaker diarisation
Jee-weon Jung, Hee-Soo Heo, Bong-Jin Lee +5
Speaker embedding extractors (EEs), which map input audio to a speaker discriminant latent space, are of paramount importance in speaker diarisation. However, there are several cha…
End-to-End Integration of Speech Recognition, Dereverberation, Beamforming, and Self-Supervised Learning Representation
Yoshiki Masuyama, Xuankai Chang, Samuele Cornell +2
Self-supervised learning representation (SSLR) has demonstrated its significant effectiveness in automatic speech recognition (ASR), mainly with clean speech. Recent work pointed o…
TF-GridNet: Making Time-Frequency Domain Models Great Again for Monaural Speaker Separation
Zhong-Qiu Wang, Samuele Cornell, Shukjae Choi +3
We propose TF-GridNet, a novel multi-path deep neural network (DNN) operating in the time-frequency (T-F) domain, for monaural talker-independent speaker separation in anechoic con…
VQ-T: RNN Transducers using Vector-Quantized Prediction Network States
Jiatong Shi, George Saon, David Haws +2
Beam search, which is the dominant ASR decoding algorithm for end-to-end models, generates tree-structured hypotheses. However, recent studies have shown that decoding with hypothe…
Improving Speech Enhancement through Fine-Grained Speech Characteristics
Muqiao Yang, Joseph Konan, David Bick +3
While deep learning based speech enhancement systems have made rapid progress in improving the quality of speech signals, they can still produce outputs that contain artifacts and…
Muskits: an End-to-End Music Processing Toolkit for Singing Voice Synthesis
Jiatong Shi, Shuai Guo, Tao Qian +9
This paper introduces a new open-source platform named Muskits for end-to-end music processing, which mainly focuses on end-to-end singing voice synthesis (E2E-SVS). Muskits suppor…
STFT-Domain Neural Speech Enhancement with Very Low Algorithmic Latency
Zhong-Qiu Wang, Gordon Wichern, Shinji Watanabe +1
Deep learning based speech enhancement in the short-time Fourier transform (STFT) domain typically uses a large window length such as 32 ms. A larger window can lead to higher freq…
Improving Frame-Online Neural Speech Enhancement with Overlapped-Frame Prediction
Zhong-Qiu Wang, Shinji Watanabe
Frame-online speech enhancement systems in the short-time Fourier transform (STFT) domain usually have an algorithmic latency equal to the window size due to the use of overlap-add…
End-to-End Integration of Speech Recognition, Speech Enhancement, and Self-Supervised Learning Representation
Xuankai Chang, Takashi Maekaku, Yuya Fujita +1
This work presents our end-to-end (E2E) automatic speech recognition (ASR) model targetting at robust speech recognition, called Integraded speech Recognition with enhanced speech…
HEAR: Holistic Evaluation of Audio Representations
Joseph Turian, Jordie Shier, Humair Raj Khan +20
What audio embedding approach generalizes best to a wide range of downstream tasks across a variety of everyday domains without fine-tuning? The aim of the HEAR benchmark is to dev…
Extended Graph Temporal Classification for Multi-Speaker End-to-End ASR
Xuankai Chang, Niko Moritz, Takaaki Hori +2
Graph-based temporal classification (GTC), a generalized form of the connectionist temporal classification loss, was recently proposed to improve automatic speech recognition (ASR)…
Discretization and Re-synthesis: an alternative method to solve the Cocktail Party Problem
Jing Shi, Xuankai Chang, Tomoki Hayashi +3
Deep learning based models have significantly improved the performance of speech separation with input mixtures like the cocktail party. Prominent methods (e.g., frequency-domain a…