788 citations · 2.9k across the 245 of their papers we have counts for
81 papers · 1 filter
Reproducing Whisper-Style Training Using an Open-Source Toolkit and Publicly Available Data
Yifan Peng, Jinchuan Tian, Brian Yan +13
Pre-training speech models on large volumes of data has achieved remarkable success. OpenAI Whisper is a multilingual multitask model trained on 680k hours of supervised speech dat…
Joint Prediction and Denoising for Large-scale Multilingual Self-supervised Learning
William Chen, Jiatong Shi, Brian Yan +6
Multilingual self-supervised learning (SSL) has often lagged behind state-of-the-art (SOTA) methods due to the expenses and complexity required to handle many languages. This furth…
Cross-Modal Multi-Tasking for Speech-to-Text Translation via Hard Parameter Sharing
Brian Yan, Xuankai Chang, Antonios Anastasopoulos +2
Recent works in end-to-end speech-to-text translation (ST) have proposed multi-tasking methods with soft parameter sharing which leverage machine translation (MT) data via secondar…
Exploring Speech Recognition, Translation, and Understanding with Discrete Speech Units: A Comparative Study
Xuankai Chang, Brian Yan, Kwanghee Choi +14
Speech signals, typically sampled at rates in the tens of thousands per second, contain redundancies, evoking inefficiencies in sequence modeling. High-dimensional speech features…
Enhancing End-to-End Conversational Speech Translation Through Target Language Context Utilization
Amir Hussein, Brian Yan, Antonios Anastasopoulos +2
Incorporating longer context has been shown to benefit machine translation, but the inclusion of context in end-to-end speech translation (E2E-ST) remains under-studied. To bridge…
BASS: Block-wise Adaptation for Speech Summarization
Roshan Sharma, Kenneth Zheng, Siddhant Arora +3
End-to-end speech summarization has been shown to improve performance over cascade baselines. However, such models are difficult to train on very large inputs (dozens of minutes or…
DPHuBERT: Joint Distillation and Pruning of Self-Supervised Speech Models
Yifan Peng, Yui Sudo, Shakeel Muhammad +1
Self-supervised learning (SSL) has achieved notable success in many speech processing tasks, but the large model size and heavy computational cost hinder the deployment. Knowledge…
A Comparative Study on E-Branchformer vs Conformer in Speech Recognition, Translation, and Understanding Tasks
Yifan Peng, Kwangyoun Kim, Felix Wu +7
Conformer, a convolution-augmented Transformer variant, has become the de facto encoder architecture for speech processing due to its superior performance in various tasks, includi…
Improving Cascaded Unsupervised Speech Translation with Denoising Back-translation
Yu-Kuan Fu, Liang-Hsuan Tseng, Jiatong Shi +4
Most of the speech translation models heavily rely on parallel data, which is hard to collect especially for low-resource languages. To tackle this issue, we propose to build a cas…
The Pipeline System of ASR and NLU with MLM-based Data Augmentation toward STOP Low-resource Challenge
Hayato Futami, Jessica Huynh, Siddhant Arora +6
This paper describes our system for the low-resource domain adaptation track (Track 3) in Spoken Language Understanding Grand Challenge, which is a part of ICASSP Signal Processing…
A Study on the Integration of Pipeline and E2E SLU systems for Spoken Semantic Parsing toward STOP Quality Challenge
Siddhant Arora, Hayato Futami, Shih-Lun Wu +6
Recently there have been efforts to introduce new benchmark tasks for spoken language understanding (SLU), like semantic parsing. In this paper, we describe our proposed spoken sem…
Joint Modelling of Spoken Language Understanding Tasks with Integrated Dialog History
Siddhant Arora, Hayato Futami, Emiru Tsunoo +2
Most human interactions occur in the form of spoken conversations where the semantic meaning of a given utterance depends on the context. Each utterance in spoken conversation can…
AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head
Rongjie Huang, Mingze Li, Dongchao Yang +10
Large language models (LLMs) have exhibited remarkable capabilities across a variety of domains and tasks, challenging our understanding of learning and cognition. Despite the rece…
I3D: Transformer architectures with input-dependent dynamic depth for speech recognition
Yifan Peng, Jaesong Lee, Shinji Watanabe
Transformer-based end-to-end speech recognition has achieved great success. However, the large footprint and computational overhead make it difficult to deploy these models in some…
Structured Pruning of Self-Supervised Pre-trained Models for Speech Recognition and Understanding
Yifan Peng, Kwangyoun Kim, Felix Wu +2
Self-supervised speech representation learning (SSL) has shown to be effective in various downstream tasks, but SSL models are usually large and slow. Model compression techniques…
Improving Massively Multilingual ASR With Auxiliary CTC Objectives
William Chen, Brian Yan, Jiatong Shi +3
Multilingual Automatic Speech Recognition (ASR) models have extended the usability of speech technologies to a wide variety of languages. With how many languages these models have…
TAPLoss: A Temporal Acoustic Parameter Loss for Speech Enhancement
Yunyang Zeng, Joseph Konan, Shuo Han +5
Speech enhancement models have greatly progressed in recent years, but still show limits in perceptual quality of their speech outputs. We propose an objective for perceptual quali…
Unsupervised Data Selection for TTS: Using Arabic Broadcast News as a Case Study
Massa Baali, Tomoki Hayashi, Hamdy Mubarak +4
Several high-resource Text to Speech (TTS) systems currently produce natural, well-established human-like speech. In contrast, low-resource languages, including Arabic, have very l…
SLUE Phase-2: A Benchmark Suite of Diverse Spoken Language Understanding Tasks
Suwon Shon, Siddhant Arora, Chyi-Jiunn Lin +7
Spoken language understanding (SLU) tasks have been studied for many decades in the speech research community, but have not received as much attention as lower-level tasks like spe…
UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Units
Hirofumi Inaguma, Sravya Popuri, Ilia Kulikov +7
Direct speech-to-speech translation (S2ST), in which all components can be optimized jointly, is advantageous over cascaded approaches to achieve fast inference with a simplified p…
Align, Write, Re-order: Explainable End-to-End Speech Translation via Operation Sequence Generation
Motoi Omachi, Brian Yan, Siddharth Dalmia +2
The black-box nature of end-to-end speech translation (E2E ST) systems makes it difficult to understand how source language inputs are being mapped to the target language. To solve…
A Study on the Integration of Pre-trained SSL, ASR, LM and SLU Models for Spoken Language Understanding
Yifan Peng, Siddhant Arora, Yosuke Higuchi +6
Collecting sufficient labeled data for spoken language understanding (SLU) is expensive and time-consuming. Recent studies achieved promising results by using pre-trained models in…
Towards Zero-Shot Code-Switched Speech Recognition
Brian Yan, Matthew Wiesner, Ondrej Klejch +2
In this work, we seek to build effective code-switched (CS) automatic speech recognition systems (ASR) under the zero-shot setting where no transcribed CS speech data is available…
Bridging Speech and Textual Pre-trained Models with Unsupervised ASR
Jiatong Shi, Chan-Jan Hsu, Holam Chung +5
Spoken language understanding (SLU) is a task aiming to extract high-level semantics from spoken utterances. Previous works have investigated the use of speech self-supervised mode…