2 citations · 6 across the 18 of their papers we have counts for
19 papers
SLVMBench: Skill Learning from Video Memory
Yudong Yang, Guangzhi Sun, Yixuan Li +1
We introduce Skill Learning from Video Memory (SLVMBench), the first benchmark that jointly evaluates whether video large language models (video-LLMs) can learn skills from long vi…
video-SALMONN-R: Learning to ReWatch, ReAsk, and ReAnswer for Efficient Video Understanding
Yixuan Li, Guangzhi Sun, Yudong Yang +1
Video large language models (LLMs) are often constrained by computation and memory budgets, leading them to use reduced frame rates and spatial resolutions, which may cause them to…
OmniMem: Perturbation-aware Memory Compression for Streaming Audio-Visual LLMs
Guangzhi Sun, Yixuan Li, Yudong Yang +1
Audio-visual large language models (LLMs) hold strong promise for long-form video understanding, yet their long-video inference is fundamentally limited by the linear growth of vid…
Learning to Attend to Depression-Related Patterns: An Adaptive Cross-Modal Gating Network for Depression Detection
Hangbin Yu, Yudong Yang, Rongfeng Su +2
Automatic depression detection using speech signals with acoustic and textual modalities is a promising approach for early diagnosis. Depression-related patterns exhibit sparsity i…
From Speech to Profile: A Protocol-Driven LLM Agent for Psychological Profile Generation
Xingjian Yang, Yudong Yang, Zhixing Guo +3
The psychological profile that structurally documents the case of a depression patient is essential for psychotherapy. Large language models can be applied to summarize the profile…
Towards General Auditory Intelligence: Large Multimodal Models for Machine Listening and Speaking
Siyin Wang, Zengrui Jin, Changli Tang +26
In the era of large language models (LLMs) and artificial general intelligence (AGI), computer audition must evolve beyond traditional paradigms to fully leverage the capabilities…
Speech-Audio Compositional Attacks on Multimodal LLMs and Their Mitigation with SALMONN-Guard
Yudong Yang, Xuezhen Zhang, Zhifeng Han +6
Recent progress in LLMs has enabled understanding of audio signals, but has also exposed new safety risks arising from complex audio inputs that are inadequately handled by current…
UTI-LLM: A Personalized Articulatory-Speech Therapy Assistance System Based on Multimodal Large Language Model
Yudong Yang, Xiaokang Liu, Shaofeng zhao +3
Speech therapy is essential for rehabilitating speech disorders caused by neurological impairments such as stroke. However, traditional manual and computer-assisted systems are lim…
video-SALMONN S: Memory-Enhanced Streaming Audio-Visual LLM
Guangzhi Sun, Yixuan Li, Xiaodong Wu +4
Long-duration streaming video understanding is fundamental for future AI agents, yet remains limited by ineffective long-term memory. We introduce video-SALMONN S, a memory-enhance…
video-SALMONN 2: Caption-Enhanced Audio-Visual Large Language Models
Changli Tang, Yixuan Li, Yudong Yang +5
We present video-SALMONN 2, a family of audio-visual large language models that set new state-of-the-art (SOTA) results in video description and question answering (QA). Our core c…
Audio-centric Video Understanding Benchmark without Text Shortcut
Yudong Yang, Jimin Zhuang, Guangzhi Sun +7
Audio often serves as an auxiliary modality in video understanding tasks of audio-visual large language models (LLMs), merely assisting in the comprehension of visual information.…
Improving LLM Video Understanding with 16 Frames Per Second
Yixuan Li, Changli Tang, Jimin Zhuang +5
Human vision is dynamic and continuous. However, in video understanding with multimodal large language models (LLMs), existing methods primarily rely on static features extracted f…
video-SALMONN-o1: Reasoning-enhanced Audio-visual Large Language Model
Guangzhi Sun, Yudong Yang, Jimin Zhuang +5
While recent advancements in reasoning optimization have significantly enhanced the capabilities of large language models (LLMs), existing efforts to improve reasoning have been li…
An End-To-End Stuttering Detection Method Based On Conformer And BILSTM
Xiaokang Liu, Changqing Xu, Yudong Yang +2
Stuttering is a neurodevelopmental speech disorder characterized by common speech symptoms such as pauses, exclamations, repetition, and prolongation. Speech-language pathologists…
Enhancing Multimodal LLM for Detailed and Accurate Video Captioning using Multi-Round Preference Optimization
Changli Tang, Yixuan Li, Yudong Yang +5
Videos contain a wealth of information, and generating detailed and accurate descriptions in natural language is a key aspect of video understanding. In this paper, we present vide…
Enabling Auditory Large Language Models for Automatic Speech Quality Evaluation
Siyin Wang, Wenyi Yu, Yudong Yang +10
Speech quality assessment typically requires evaluating audio from multiple aspects, such as mean opinion score (MOS) and speaker similarity (SIM) \etc., which can be challenging t…
Extract and Diffuse: Latent Integration for Improved Diffusion-based Speech and Vocal Enhancement
Yudong Yang, Zhan Liu, Wenyi Yu +3
Diffusion-based generative models have recently achieved remarkable results in speech and vocal enhancement due to their ability to model complex speech data distributions. While t…
Automatic Assessment of Dysarthria Using Audio-visual Vowel Graph Attention Network
Xiaokang Liu, Xiaoxia Du, Juan Liu +7
Automatic assessment of dysarthria remains a highly challenging task due to high variability in acoustic signals and the limited data. Currently, research on the automatic assessme…
An Audio-textual Diffusion Model For Converting Speech Signals Into Ultrasound Tongue Imaging Data
Yudong Yang, Rongfeng Su, Xiaokang Liu +2
Acoustic-to-articulatory inversion (AAI) is to convert audio into articulator movements, such as ultrasound tongue imaging (UTI) data. An issue of existing AAI methods is only usin…