7 papers
Time-JEPA: Grounding Video World Models with Physical Time Series
Shenyi Li, Ziyang Zhang, Wen Wu +2
Video world models learn how visual environments evolve by predicting missing or future content. However, physical-state information can provide useful context for predicting how e…
TVPara: Paralinguistic Captioning of Expressive TV-Series Speech in Complex Acoustic Environments
Ziyang Zhang, Wenyi Yu, Wen Wu +2
Large audio-language models (LALMs) struggle to parse paralinguistics (e.g., emotion, pace, pitch) in speech that combines intense emotion with complex acoustic environments. Stren…
SALMONN-2: Advancing General-Purpose Hearing Abilities with Self-Supervised Representations
Xiaoyu Yang, Xuenan Xu, Wenyi Yu +10
Recent audio large language models (ALLMs) are typically built upon audio encoders trained with large amounts of supervised data. Since self-supervised learning (SSL) audio encoder…
AuDirector: A Self-Reflective Closed-Loop Framework for Immersive Audio Storytelling
Yiming Ren, Xuenan Xu, Ziyang Zhang +3
Despite advances in text and visual generation, creating coherent long-form audio narratives remains challenging. Existing frameworks often exhibit limitations such as mismatched c…
Bayesian Speech Synthesizers Can Learn from Multiple Teachers
Ziyang Zhang, Yifan Gao, Xuenan Xu +3
Text-to-speech (TTS) is a "one-to-many" problem: one text can be spoken in many valid ways, so its realizations form a distribution whose spread depends on the input. Most TTS mode…
SWIM: Short-Window CNN Integrated with Mamba for EEG-Based Auditory Spatial Attention Decoding
Ziyang Zhang, Andrew Thwaites, Alexandra Woolgar +2
In complex auditory environments, the human auditory system possesses the remarkable ability to focus on a specific speaker while disregarding others. In this study, a new model na…
Bayesian WeakS-to-Strong from Text Classification to Generation
Ziyun Cui, Ziyang Zhang, Guangzhi Sun +2
Advances in large language models raise the question of how alignment techniques will adapt as models become increasingly complex and humans will only be able to supervise them wea…