30 citations · 41 across the 22 of their papers we have counts for
22 papers
Spoken Language Models that Think Aloud
Junyi Ao, Kainan Peng, Mingbo Ma +10
While Chain-of-Thought (CoT) reasoning has improved the capability of language models, directly applying it to Spoken Language Models (SLMs) may introduce long silent intervals und…
What Did I Just Say? Self-Listening for Full-Duplex Speech Models
Xuanning Zhou, Junyi Ao, Xiaotong Liu +3
Full-duplex spoken language models can listen and speak simultaneously, enabling them to handle interruptions and backchannels in human conversation. However, text generation, spee…
Scaling Speech Tokenizers with Diffusion Autoencoders
Yuancheng Wang, Zhenyu Tang, Yun Wang +9
Speech tokenizers are foundational to speech language models, yet existing approaches face two major challenges: (1) balancing trade-offs between encoding semantics for understandi…
Leveraging Language Information for Target Language Extraction
Mehmet Sinan Yıldırım, Ruijie Tao, Wupeng Wang +2
Target Language Extraction aims to extract speech in a specific language from a mixture waveform that contains multiple speakers speaking different languages. The human auditory sy…
EchoMind: An Interrelated Multi-level Benchmark for Evaluating Empathetic Speech Language Models
Li Zhou, Lutong Yu, You Lyu +6
Speech Language Models (SLMs) have made significant progress in spoken language understanding. Yet it remains unclear whether they can fully perceive non lexical vocal cues alongsi…
Audio Deepfake Verification
Li Wang, Junyi Ao, Linyong Gan +3
With the rapid development of deepfake technology, simply making a binary judgment of true or false on audio is no longer sufficient to meet practical needs. Accurately determining…
Solla: Towards a Speech-Oriented LLM That Hears Acoustic Context
Junyi Ao, Dekun Chen, Xiaohai Tian +6
Large Language Models (LLMs) have recently shown remarkable ability to process not only text but also multimodal inputs such as speech and audio. However, most existing models prim…
Overview of the Amphion Toolkit (v0.2)
Jiaqi Li, Xueyao Zhang, Yuancheng Wang +9
Amphion is an open-source toolkit for Audio, Music, and Speech Generation, designed to lower the entry barrier for junior researchers and engineers in these fields. It provides a v…
SA-WavLM: Speaker-Aware Self-Supervised Pre-training for Mixture Speech
Jingru Lin, Meng Ge, Junyi Ao +2
It was shown that pre-trained models with self-supervised learning (SSL) techniques are effective in various downstream speech tasks. However, most such models are trained on singl…
SD-Eval: A Benchmark Dataset for Spoken Dialogue Understanding Beyond Words
Junyi Ao, Yuancheng Wang, Xiaohai Tian +6
Speech encompasses a wealth of information, including but not limited to content, paralinguistic, and environmental information. This comprehensive nature of speech significantly i…
Text-guided HuBERT: Self-Supervised Speech Pre-training via Generative Adversarial Networks
Duo Ma, Xianghu Yue, Junyi Ao +2
Human language can be expressed in either written or spoken form, i.e. text or speech. Humans can acquire knowledge from text to improve speaking and listening. However, the quest…
The NUS-HLT System for ICASSP2024 ICMC-ASR Grand Challenge
Meng Ge, Yizhou Peng, Yidi Jiang +6
This paper summarizes our team's efforts in both tracks of the ICMC-ASR Challenge for in-car multi-channel automatic speech recognition. Our submitted systems for ICMC-ASR Challeng…
USED: Universal Speaker Extraction and Diarization
Junyi Ao, Mehmet Sinan Yıldırım, Ruijie Tao +4
Speaker extraction and diarization are two enabling techniques for real-world speech applications. Speaker extraction aims to extract a target speaker's voice from a speech mixture…
Self-Supervised Acoustic Word Embedding Learning via Correspondence Transformer Encoder
Jingru Lin, Xianghu Yue, Junyi Ao +1
Acoustic word embeddings (AWEs) aims to map a variable-length speech segment into a fixed-dimensional representation. High-quality AWEs should be invariant to variations, such as d…
token2vec: A Joint Self-Supervised Pre-training Framework Using Unpaired Speech and Text
Xianghu Yue, Junyi Ao, Xiaoxue Gao +1
Self-supervised pre-training has been successful in both text and speech processing. Speech and text offer different but complementary information. The question is whether we are a…
SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-training
Ziqiang Zhang, Long Zhou, Junyi Ao +4
The rapid development of single-modal pre-training has prompted researchers to pay more attention to cross-modal pre-training methods. In this paper, we propose a unified-modal spe…
CoBERT: Self-Supervised Speech Representation Learning Through Code Representation Learning
Chutong Meng, Junyi Ao, Tom Ko +2
Speech is the surface form of a finite set of phonetic units, which can be represented by discrete codes. We propose the Code BERT (CoBERT) approach for self-supervised speech repr…
The YiTrans End-to-End Speech Translation System for IWSLT 2022 Offline Shared Task
Ziqiang Zhang, Junyi Ao, Long Zhou +3
This paper describes the submission of our end-to-end YiTrans speech translation system for the IWSLT 2022 offline task, which translates from English audio to German, Chinese, and…
Pre-Training Transformer Decoder for End-to-End ASR Model with Unpaired Speech Data
Junyi Ao, Ziqiang Zhang, Long Zhou +7
This paper studies a novel pre-training technique with unpaired speech data, Speech2C, for encoder-decoder based automatic speech recognition (ASR). Within a multi-task learning fr…
LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERT
Rui Wang, Qibing Bai, Junyi Ao +6
Self-supervised speech representation learning has shown promising results in various speech processing tasks. However, the pre-trained models, e.g., HuBERT, are storage-intensive…
SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing
Junyi Ao, Rui Wang, Long Zhou +11
Motivated by the success of T5 (Text-To-Text Transfer Transformer) in pre-trained natural language processing models, we propose a unified-modal SpeechT5 framework that explores th…
Multi-View Self-Attention Based Transformer for Speaker Recognition
Rui Wang, Junyi Ao, Long Zhou +5
Initially developed for natural language processing (NLP), Transformer model is now widely used for speech processing tasks such as speaker recognition, due to its powerful sequenc…