Publications (102)
HuBERTopic: Enhancing Semantic Representation of HuBERT through Self-supervision Utilizing Topic Model
Takashi Maekaku, Jiatong Shi, Xuankai Chang +2
SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learning
Tzu-hsun Feng, Annie Dong, Ching-Feng Yeh +11
FoodPuzzle: Developing Large Language Model Agents as Flavor Scientists
Tenghao Huang, Donghee Lee, John Sweeney +5
AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head
Rongjie Huang, Mingze Li, Dongchao Yang +10
Robust Training of Singing Voice Synthesis Using Prior and Posterior Uncertainty
Yiwen Zhao, Jiatong Shi, Yuxun Tang +2
Improving Cascaded Unsupervised Speech Translation with Denoising Back-translation
Yu-Kuan Fu, Liang-Hsuan Tseng, Jiatong Shi +4
UniAudio: An Audio Foundation Model Toward Universal Audio Generation
Dongchao Yang, Jinchuan Tian, Xu Tan +9
VISinger2+: End-to-End Singing Voice Synthesis Augmented by Self-Supervised Learning Representation
Yifeng Yu, Jiatong Shi, Yuning Wu +2
SUPERB: Speech processing Universal PERformance Benchmark
Shu-wen Yang, Po-Han Chi, Yung-Sung Chuang +17
The Interspeech 2024 Challenge on Speech Processing Using Discrete Units
Xuankai Chang, Jiatong Shi, Jinchuan Tian +7
Enhancing Speech-to-Speech Translation with Multiple TTS Targets
Jiatong Shi, Yun Tang, Ann Lee +4
Blockwise Streaming Transformer for Spoken Language Understanding and Simultaneous Speech Translation
Keqi Deng, Shinji Watanabe, Jiatong Shi +1
Understanding the Tradeoffs in Client-side Privacy for Downstream Speech Tasks
Peter Wu, Paul Pu Liang, Jiatong Shi +3
The ML-SUPERB 2.0 Challenge: Towards Inclusive ASR Benchmarking for All Language Varieties
William Chen, Chutong Meng, Jiatong Shi +10
Discrete Audio Tokens: More Than a Survey!
Pooneh Mousavi, Gallil Maimon, Adel Moumen +18
Balancing Speech Understanding and Generation Using Continual Pre-training for Codec-based Speech LLM
Jiatong Shi, Chunlei Zhang, Jinchuan Tian +4
Fusion of Discrete Representations and Self-Augmented Representations for Multilingual Automatic Speech Recognition
Shih-heng Wang, Jiatong Shi, Chien-yu Huang +2
OpusLM: A Family of Open Unified Speech Language Models
Jinchuan Tian, William Chen, Yifan Peng +9
ESPnet-SDS: Unified Toolkit and Demo for Spoken Dialogue Systems
Siddhant Arora, Yifan Peng, Jiatong Shi +9
Adapting Speech Language Model to Singing Voice Synthesis
Yiwen Zhao, Jiatong Shi, Jinchuan Tian +4
MMM: Multi-Layer Multi-Residual Multi-Stream Discrete Speech Representation from Self-supervised Learning Model
Jiatong Shi, Xutai Ma, Hirofumi Inaguma +2
ESPnet-ST-v2: Multipurpose Spoken Language Translation Toolkit
Brian Yan, Jiatong Shi, Yun Tang +13
VERSA: A Versatile Evaluation Toolkit for Speech, Audio, and Music
Jiatong Shi, Hye-jin Shim, Jinchuan Tian +14
4D ASR: Joint modeling of CTC, Attention, Transducer, and Mask-Predict decoders
Yui Sudo, Muhammad Shakeel, Brian Yan +2
Exploring Speech Recognition, Translation, and Understanding with Discrete Speech Units: A Comparative Study
Xuankai Chang, Brian Yan, Kwanghee Choi +14
ANCHOR: Autoregressive Non-intrusive Chunk-Ordered Refinement for Joint Multi-Resolution Speech Quality Modeling
Zhuoyan Tao, Jiatong Shi, Hye-jin Shim +1
Chain-of-Thought Training for Open E2E Spoken Dialogue Systems
Siddhant Arora, Jinchuan Tian, Hayato Futami +5
Findings of the IWSLT 2024 Evaluation Campaign
Ibrahim Said Ahmad, Antonios Anastasopoulos, OndÅej Bojar +42
An Extensive Analysis of the Singing Voice Conversion Challenge 2025 Evaluation Results
Lester Phillip Violeta, Xueyao Zhang, Jiatong Shi +4
Cross-lingual Transfer for Speech Processing using Acoustic Language Similarity
Peter Wu, Jiatong Shi, Yifan Zhong +2
The Singing Voice Conversion Challenge 2023
Wen-Chin Huang, Lester Phillip Violeta, Songxiang Liu +2
ESPnet-ST IWSLT 2021 Offline Speech Translation System
Hirofumi Inaguma, Brian Yan, Siddharth Dalmia +4
ESPnet3: Infrastructure for Scalable Speech and Audio Research in the Foundation Model Era
Masao Someki, Alexander Polok, Carlos Carvalho +14
SingingSDS: A Singing-Capable Spoken Dialogue System for Conversational Roleplay Applications
Jionghao Han, Jiatong Shi, Masao Someki +5
Sequence-to-sequence Singing Voice Synthesis with Perceptual Entropy Loss
Jiatong Shi, Shuai Guo, Nan Huo +2
How to Learn a New Language? An Efficient Solution for Self-Supervised Learning Models Unseen Languages Adaption in Low-Resource Scenario
Shih-Heng Wang, Zih-Ching Chen, Jiatong Shi +6
SVDD Challenge 2024: A Singing Voice Deepfake Detection Challenge Evaluation Plan
You Zhang, Yongyi Zang, Jiatong Shi +5
IKFST: IOO and KOO Algorithms for Accelerated and Precise WFST-based End-to-End Automatic Speech Recognition
Zhuoran Zhuang, Ye Chen, Chao Luo +5
SVDD 2024: The Inaugural Singing Voice Deepfake Detection Challenge
You Zhang, Yongyi Zang, Jiatong Shi +3
SingAug: Data Augmentation for Singing Voice Synthesis with Cycle-consistent Training Strategy
Shuai Guo, Jiatong Shi, Tao Qian +2
Do Neural Codecs Generalize? A Controlled Study Across Unseen Languages and Non-Speech Tasks
Shih-Heng Wang, Jiatong Shi, Jinchuan Tian +2
Reasoning Beyond Majority Vote: An Explainable SpeechLM Framework for Speech Emotion Recognition
Bo-Hao Su, Hui-Ying Shih, Jinchuan Tian +4
Joint Prediction and Denoising for Large-scale Multilingual Self-supervised Learning
William Chen, Jiatong Shi, Brian Yan +6
Joint Beam Search Integrating CTC, Attention, and Transducer Decoders
Yui Sudo, Muhammad Shakeel, Yosuke Fukumoto +4
Exploiting Longitudinal Speech Sessions via Voice Assistant Systems for Early Detection of Cognitive Decline
Kristin Qi, Jiatong Shi, Caroline Summerour +2
Exploration on HuBERT with Multiple Resolutions
Jiatong Shi, Yun Tang, Hirofumi Inaguma +3
ARECHO: Autoregressive Evaluation via Chain-Based Hypothesis Optimization for Speech Multi-Metric Estimation
Jiatong Shi, Yifan Cheng, Bo-Hao Su +6
Dynamic-SUPERB: Towards A Dynamic, Collaborative, and Comprehensive Instruction-Tuning Benchmark for Speech
Chien-yu Huang, Ke-Han Lu, Shih-Heng Wang +12
Chain-of-Thought Reasoning in Streaming Full-Duplex End-to-End Spoken Dialogue Systems
Siddhant Arora, Jinchuan Tian, Hayato Futami +4
A Systematic Exploration of Joint-training for Singing Voice Synthesis
Yuning Wu, Yifeng Yu, Jiatong Shi +2
PURE Codec: Progressive Unfolding of Residual Entropy for Speech Codec Learning
Jiatong Shi, Haoran Wang, William Chen +4
CtrSVDD: A Benchmark Dataset and Baseline Analysis for Controlled Singing Voice Deepfake Detection
Yongyi Zang, Jiatong Shi, You Zhang +8
ESPnet-SPK: full pipeline speaker embedding toolkit with reproducible recipes, self-supervised front-ends, and off-the-shelf models
Jee-weon Jung, Wangyou Zhang, Jiatong Shi +5
ML-SUPERB 2.0: Benchmarking Multilingual Speech Models Across Modeling Constraints, Languages, and Datasets
Jiatong Shi, Shih-Heng Wang, William Chen +8
Full-Duplex-Bench-v2: A Multi-Turn Evaluation Framework for Duplex Dialogue Systems with an Automated Examiner
Guan-Ting Lin, Shih-Yun Shan Kuan, Jiatong Shi +4
On Compressing Sequences for Self-Supervised Speech Models
Yen Meng, Hsuan-Jui Chen, Jiatong Shi +4
MMGenre: Benchmarking Singing Voice Synthesis across Multiple Musical Genres
Wenhao Feng, Yuxun Tang, Jiatong Shi +1
ESPnet-EZ: Python-only ESPnet for Easy Fine-tuning and Integration
Masao Someki, Kwanghee Choi, Siddhant Arora +7
Towards Robust Speech Representation Learning for Thousands of Languages
William Chen, Wangyou Zhang, Yifan Peng +7
Recent Developments on ESPnet Toolkit Boosted by Conformer
Pengcheng Guo, Florian Boyer, Xuankai Chang +12
Wav2Gloss: Generating Interlinear Glossed Text from Speech
Taiqi He, Kwanghee Choi, Lindia Tjuatja +6
Improving Massively Multilingual ASR With Auxiliary CTC Objectives
William Chen, Brian Yan, Jiatong Shi +3
Optimizing Conversational Quality in Spoken Dialogue Systems with Reinforcement Learning from AI Feedback
Siddhant Arora, Jinchuan Tian, Jiatong Shi +4
PHONEix: Acoustic Feature Processing Strategy for Enhanced Singing Pronunciation with Phoneme Distribution Predictor
Yuning Wu, Jiatong Shi, Tao Qian +2
Self-supervised Speech Representations Still Struggle with African American Vernacular English
Kalvin Chang, Yi-Hui Chou, Jiatong Shi +4
SingMOS: An extensive Open-Source Singing Voice Dataset for MOS Prediction
Yuxun Tang, Jiatong Shi, Yuning Wu +1
CartoonSing: Unifying Human and Nonhuman Timbres in Singing Generation
Jionghao Han, Jiatong Shi, Zhuoyan Tao +4
Findings of the 2023 ML-SUPERB Challenge: Pre-Training and Evaluation over More Languages and Beyond
Jiatong Shi, William Chen, Dan Berrebbi +10
ESPnet2-TTS: Extending the Edge of TTS Research
Tomoki Hayashi, Ryuichi Yamamoto, Takenori Yoshimura +7
Improving RNN Transducer With Target Speaker Extraction and Neural Uncertainty Estimation
Jiatong Shi, Chunlei Zhang, Chao Weng +3
SingOMD: Singing Oriented Multi-resolution Discrete Representation Construction from Speech Models
Yuxun Tang, Yuning Wu, Jiatong Shi +1
SingMOS-Pro: An Comprehensive Benchmark for Singing Quality Assessment
Yuxun Tang, Lan Liu, Wenhao Feng +5
MIKU-PAL: An Automated and Standardized Multi-Modal Method for Speech Paralinguistic and Affect Labeling
Yifan Cheng, Ruoyi Zhang, Jiatong Shi
Multi-resolution HuBERT: Multi-resolution Speech Self-Supervised Learning with Masked Unit Prediction
Jiatong Shi, Hirofumi Inaguma, Xutai Ma +2
Leveraging End-to-End ASR for Endangered Language Documentation: An Empirical Study on Yoloxóchitl Mixtec
Jiatong Shi, Jonathan D. Amith, Rey Castillo GarcÃa +3
ESPnet-Codec: Comprehensive Training and Evaluation of Neural Codecs for Audio, Music, and Speech
Jiatong Shi, Jinchuan Tian, Yihan Wu +17
Bridging Speech and Textual Pre-trained Models with Unsupervised ASR
Jiatong Shi, Chan-Jan Hsu, Holam Chung +5
Muskits-ESPnet: A Comprehensive Toolkit for Singing Voice Synthesis in New Paradigm
Yuning Wu, Jiatong Shi, Yifeng Yu +7
SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilities
Hsiang-Sheng Tsai, Heng-Jui Chang, Wen-Chin Huang +14
ESPnet-SpeechLM: An Open Speech Language Model Toolkit
Jinchuan Tian, Jiatong Shi, William Chen +13
EFFUSE: Efficient Self-Supervised Feature Fusion for E2E ASR in Low Resource and Multilingual Scenarios
Tejes Srivastava, Jiatong Shi, William Chen +1
Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark for Measuring the Capabilities of Spoken Language Models with 180 Tasks
Chien-yu Huang, Wei-Chih Chen, Shu-wen Yang +77
Evaluating Self-supervised Speech Models on a Taiwanese Hokkien Corpus
Yi-Hui Chou, Kalvin Chang, Meng-Ju Wu +12
Uni-VERSA: Versatile Speech Assessment with a Unified Network
Jiatong Shi, Hye-Jin Shim, Shinji Watanabe
Improving Speech Enhancement with Multi-Metric Supervision from Learned Quality Assessment
Wei Wang, Wangyou Zhang, Chenda Li +3
DiscoSum: Discourse-aware News Summarization
Alexander Spangher, Tenghao Huang, Jialiang Gu +2
VQ-T: RNN Transducers using Vector-Quantized Prediction Network States
Jiatong Shi, George Saon, David Haws +2
BSCodec: A Band-Split Neural Codec for High-Quality Universal Audio Reconstruction
Haoran Wang, Jiatong Shi, Jinchuan Tian +3
Muskits: an End-to-End Music Processing Toolkit for Singing Voice Synthesis
Jiatong Shi, Shuai Guo, Tao Qian +9
Speech-DRAME: A Framework for Human-Aligned Benchmarks in Speech Role-Play
Jiatong Shi, Jionghao Han, Yichen Lu +6
Reproducing Whisper-Style Training Using an Open-Source Toolkit and Publicly Available Data
Yifan Peng, Jinchuan Tian, Brian Yan +13
Combining Spectral and Self-Supervised Features for Low Resource Speech Recognition and Translation
Dan Berrebbi, Jiatong Shi, Brian Yan +3
OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on E-Branchformer
Yifan Peng, Jinchuan Tian, William Chen +9
Singing Voice Data Scaling-up: An Introduction to ACE-Opencpop and ACE-KiSing
Jiatong Shi, Yueqian Lin, Xinyi Bai +6
Aligning Text-to-Music Evaluation with Human Preferences
Yichen Huang, Zachary Novack, Koichi Saito +5
EURO: ESPnet Unsupervised ASR Open-source Toolkit
Dongji Gao, Jiatong Shi, Shun-Po Chuang +4
ML-SUPERB: Multilingual Speech Universal PERformance Benchmark
Jiatong Shi, Dan Berrebbi, William Chen +8
TokSing: Singing Voice Synthesis based on Discrete Tokens
Yuning Wu, Chunlei zhang, Jiatong Shi +3
Context-aware Goodness of Pronunciation for Computer-Assisted Pronunciation Training
Jiatong Shi, Nan Huo, Qin Jin
Preference Alignment Improves Language Model-Based TTS
Jinchuan Tian, Chunlei Zhang, Jiatong Shi +4