14 citations · 17 across the 11 of their papers we have counts for
12 papers
SingingSDS: A Singing-Capable Spoken Dialogue System for Conversational Roleplay Applications
Jionghao Han, Jiatong Shi, Masao Someki +5
With recent advances in automatic speech recognition (ASR), large language models (LLMs), and text-to-speech (TTS) technologies, spoken dialogue systems (SDS) have become widely ac…
Adapting Speech Language Model to Singing Voice Synthesis
Yiwen Zhao, Jiatong Shi, Jinchuan Tian +4
Speech Language Models (SLMs) have recently emerged as a unified paradigm for addressing a wide range of speech-related tasks, including text-to-speech (TTS), speech enhancement (S…
CartoonSing: Unifying Human and Nonhuman Timbres in Singing Generation
Jionghao Han, Jiatong Shi, Zhuoyan Tao +4
Singing voice synthesis (SVS) and singing voice conversion (SVC) have achieved remarkable progress in generating natural-sounding human singing. However, existing systems are restr…
Speech-DRAME: A Framework for Human-Aligned Benchmarks in Speech Role-Play
Jiatong Shi, Jionghao Han, Yichen Lu +6
Role-play has become a key testbed for generative models, expanding from text-only dialogue to multimodal interaction. Extending role-play to speech captures prosody, emotion, and…
SingMOS-Pro: An Comprehensive Benchmark for Singing Quality Assessment
Yuxun Tang, Lan Liu, Wenhao Feng +5
Singing voice generation progresses rapidly, yet evaluating singing quality remains a critical challenge. Human subjective assessment, typically in the form of listening tests, is…
VERSA: A Versatile Evaluation Toolkit for Speech, Audio, and Music
Jiatong Shi, Hye-jin Shim, Jinchuan Tian +14
In this work, we introduce VERSA, a unified and standardized evaluation toolkit designed for various speech, audio, and music signals. The toolkit features a Pythonic interface wit…
Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark for Measuring the Capabilities of Spoken Language Models with 180 Tasks
Chien-yu Huang, Wei-Chih Chen, Shu-wen Yang +77
Multimodal foundation models, such as Gemini and ChatGPT, have revolutionized human-machine interactions by seamlessly integrating various forms of data. Developing a universal spo…
Muskits-ESPnet: A Comprehensive Toolkit for Singing Voice Synthesis in New Paradigm
Yuning Wu, Jiatong Shi, Yifeng Yu +7
This research presents Muskits-ESPnet, a versatile toolkit that introduces new paradigms to Singing Voice Synthesis (SVS) through the application of pretrained audio models in both…
ESPnet-EZ: Python-only ESPnet for Easy Fine-tuning and Integration
Masao Someki, Kwanghee Choi, Siddhant Arora +7
We introduce ESPnet-EZ, an extension of the open-source speech processing toolkit ESPnet, aimed at quick and easy development of speech models. ESPnet-EZ focuses on two major aspec…
CtrSVDD: A Benchmark Dataset and Baseline Analysis for Controlled Singing Voice Deepfake Detection
Yongyi Zang, Jiatong Shi, You Zhang +8
Recent singing voice synthesis and conversion advancements necessitate robust singing voice deepfake detection (SVDD) models. Current SVDD datasets face challenges due to limited c…
On the Evaluation of Speech Foundation Models for Spoken Language Understanding
Siddhant Arora, Ankita Pasad, Chung-Ming Chien +9
The Spoken Language Understanding Evaluation (SLUE) suite of benchmark tasks was recently introduced to address the need for open resources and benchmarking of complex spoken langu…
SVDD Challenge 2024: A Singing Voice Deepfake Detection Challenge Evaluation Plan
You Zhang, Yongyi Zang, Jiatong Shi +5
The rapid advancement of AI-generated singing voices, which now closely mimic natural human singing and align seamlessly with musical scores, has led to heightened concerns for art…