18 citations · 47 across the 61 of their papers we have counts for
11 papers · 1 filter
VERSA: A Versatile Evaluation Toolkit for Speech, Audio, and Music
Jiatong Shi, Hye-jin Shim, Jinchuan Tian +14
In this work, we introduce VERSA, a unified and standardized evaluation toolkit designed for various speech, audio, and music signals. The toolkit features a Pythonic interface wit…
Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark for Measuring the Capabilities of Spoken Language Models with 180 Tasks
Chien-yu Huang, Wei-Chih Chen, Shu-wen Yang +77
Multimodal foundation models, such as Gemini and ChatGPT, have revolutionized human-machine interactions by seamlessly integrating various forms of data. Developing a universal spo…
Hypothesis Clustering and Merging: Novel MultiTalker Speech Recognition with Speaker Tokens
Yosuke Kashiwagi, Hayato Futami, Emiru Tsunoo +2
In many real-world scenarios, such as meetings, multiple speakers are present with an unknown number of participants, and their utterances often overlap. We address these multi-spe…
ESPnet-EZ: Python-only ESPnet for Easy Fine-tuning and Integration
Masao Someki, Kwanghee Choi, Siddhant Arora +7
We introduce ESPnet-EZ, an extension of the open-source speech processing toolkit ESPnet, aimed at quick and easy development of speech models. ESPnet-EZ focuses on two major aspec…
Task Arithmetic for Language Expansion in Speech Translation
Yao-Fei Cheng, Hayato Futami, Yosuke Kashiwagi +4
Recent progress in large language models (LLMs) has gained interest in speech-text multimodal foundation models, achieving strong performance on instruction-tuned speech translatio…
Decoder-only Architecture for Streaming End-to-end Speech Recognition
Emiru Tsunoo, Hayato Futami, Yosuke Kashiwagi +2
Decoder-only language models (LMs) have been successfully adopted for speech-processing tasks including automatic speech recognition (ASR). The LMs have ample expressiveness and pe…
Rapid Language Adaptation for Multilingual E2E Speech Recognition Using Encoder Prompting
Yosuke Kashiwagi, Hayato Futami, Emiru Tsunoo +2
End-to-end multilingual speech recognition models handle multiple languages through a single model, often incorporating language identification to automatically detect the language…
Finding Task-specific Subnetworks in Multi-task Spoken Language Understanding Model
Hayato Futami, Siddhant Arora, Yosuke Kashiwagi +2
Recently, multi-task spoken language understanding (SLU) models have emerged, designed to address various speech processing tasks. However, these models often rely on a large numbe…
On the Evaluation of Speech Foundation Models for Spoken Language Understanding
Siddhant Arora, Ankita Pasad, Chung-Ming Chien +9
The Spoken Language Understanding Evaluation (SLUE) suite of benchmark tasks was recently introduced to address the need for open resources and benchmarking of complex spoken langu…
TMT: Tri-Modal Translation between Speech, Image, and Text by Processing Different Modalities as Different Languages
Minsu Kim, Jee-weon Jung, Hyeongseop Rha +5
The capability to jointly process multi-modal information is becoming an essential task. However, the limited number of paired multi-modal data and the large computational requirem…
OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on E-Branchformer
Yifan Peng, Jinchuan Tian, William Chen +9
Recent studies have highlighted the importance of fully open foundation models. The Open Whisper-style Speech Model (OWSM) is an initial step towards reproducing OpenAI Whisper usi…