3 citations · 6 across the 15 of their papers we have counts for
17 papers
-Bench: Evaluating Speech Interaction Models as Scientific Voice Assistants
Heyang Liu, Jiayi Huang, Wenyang Xiao +8
The advance of multimodal large language models (MLLMs) has fundamentally reshaped the paradigm of human-computer interaction, especially speech interaction models capable of seaml…
LaSR: Context-Aware Speech Recognition via Latent Reasoning
Heyang Liu, Ziyang Cheng, Jiayi Huang +5
Speech recognition in specialized domains requires leveraging contextual or topical information to improve the recognition of domain-specific entities. Speech Large Language Models…
VocalNet-MDM: Accelerating Streaming Speech LLM via Self-Distilled Masked Diffusion Modeling
Ziyang Cheng, Yuhao Wang, Heyang Liu +4
Recent Speech Large Language Models~(LLMs) have achieved impressive capabilities in end-to-end speech interaction. However, the prevailing autoregressive paradigm imposes strict se…
VocalBench-zh: Decomposing and Benchmarking the Speech Conversational Abilities in Mandarin Context
Heyang Liu, Ziyang Cheng, Yuhao Wang +6
The development of multi-modal large language models (LLMs) leads to intelligent approaches capable of speech interactions. As one of the most widely spoken languages globally, Man…
VocalNet-M2: Advancing Low-Latency Spoken Language Modeling via Integrated Multi-Codebook Tokenization and Multi-Token Prediction
Yuhao Wang, Ziyang Cheng, Heyang Liu +4
Current end-to-end spoken language models (SLMs) have made notable progress, yet they still encounter considerable response latency. This delay primarily arises from the autoregres…
VocalBench-DF: A Benchmark for Evaluating Speech LLM Robustness to Disfluency
Hongcheng Liu, Yixuan Hou, Heyang Liu +3
While Speech Large Language Models (Speech-LLMs) show strong performance in many applications, their robustness is critically under-tested, especially to speech disfluency. Existin…
CS3-Bench: Evaluating and Enhancing Speech-to-Speech LLMs for Mandarin-English Code-Switching
Heyang Liu, Yuhao Wang, Ziyang Cheng +4
The advancement of multimodal large language models has accelerated the development of speech-to-speech interaction systems. While natural monolingual interaction has been achieved…
SOVA-Bench: Benchmarking the Speech Conversation Ability for LLM-based Voice Assistant
Yixuan Hou, Heyang Liu, Yuhao Wang +5
Thanks to the steady progress of large language models (LLMs), speech encoding algorithms and vocoder structure, recent advancements have enabled generating speech response directl…
VocalNet: Speech LLM with Multi-Token Prediction for Faster and High-Quality Generation
Yuhao Wang, Heyang Liu, Ziyang Cheng +4
Speech large language models (LLMs) have emerged as a prominent research focus in speech processing. We introduce VocalNet-1B and VocalNet-8B, a series of high-performance, low-lat…
Drawing the Line: Enhancing Trustworthiness of MLLMs Through the Power of Refusal
Yuhao Wang, Zhiyuan Zhu, Heyang Liu +4
Multimodal large language models (MLLMs) excel at multimodal perception and understanding, yet their tendency to generate hallucinated or inaccurate responses undermines their trus…
Med-PMC: Medical Personalized Multi-modal Consultation with a Proactive Ask-First-Observe-Next Paradigm
Hongcheng Liu, Yusheng Liao, Siqv Ou +4
The application of the Multi-modal Large Language Models (MLLMs) in medical clinical scenarios remains underexplored. Previous benchmarks only focus on the capacity of the MLLMs in…
Decoding Linguistic Representations of Human Brain
Yu Wang, Heyang Liu, Yuhao Wang +6
Language, as an information medium created by advanced organisms, has always been a concern of neuroscience regarding how it is represented in the brain. Decoding linguistic repres…
Towards an End-to-End Framework for Invasive Brain Signal Decoding with Large Language Models
Sheng Feng, Heyang Liu, Yu Wang +1
In this paper, we introduce a groundbreaking end-to-end (E2E) framework for decoding invasive brain signals, marking a significant advancement in the field of speech neuroprosthesi…
MAV: A Multimodal, Multigenre, and Multipurpose Audio-Visual Academic Lecture Dataset
Zhe Chen, Heyang Liu, Wenyi Yu +6
Publishing open-source academic video recordings is an emergent and prevalent approach to sharing knowledge online. Such videos carry rich multimodal information including speech,…
Post-decoder Biasing for End-to-End Speech Recognition of Multi-turn Medical Interview
Heyang Liu, Yu Wang, Yanfeng Wang
End-to-end (E2E) approach is gradually replacing hybrid models for automatic speech recognition (ASR) tasks. However, the optimization of E2E models lacks an intuitive method for h…
MM-SAP: A Comprehensive Benchmark for Assessing Self-Awareness of Multimodal Large Language Models in Perception
Yuhao Wang, Yusheng Liao, Heyang Liu +3
Recent advancements in Multimodal Large Language Models (MLLMs) have demonstrated exceptional capabilities in visual perception and understanding. However, these models also suffer…
LibriSQA: A Novel Dataset and Framework for Spoken Question Answering with Large Language Models
Zihan Zhao, Yiyang Jiang, Heyang Liu +2
While Large Language Models (LLMs) have demonstrated commendable performance across a myriad of domains and tasks, existing LLMs still exhibit a palpable deficit in handling multim…