2 citations · 5 across the 17 of their papers we have counts for
17 papers
How Much Future Helps? A Controlled Study of Future-Privileged Supervision for Causal Egocentric Gaze Estimation
Jia Li, Wenjie Zhao, Fnu Atisri +5
Egocentric gaze estimation is commonly studied using models that process the full video with access to future frames, while real-world applications require strictly causal, online…
Do Joint Audio-Video Generation Models Understand Physics?
Zijun Cui, Xiulong Liu, Hao Fang +8
Joint audio-video generation models are rapidly approaching professional production quality, raising a central question: do they understand audio-visual physics, or merely generate…
Audio-Visual Intelligence in Large Foundation Models
You Qin, Kai Liu, Shengqiong Wu +12
Audio-Visual Intelligence (AVI) has emerged as a central frontier in artificial intelligence, bridging auditory and visual modalities to enable machines that can perceive, generate…
OmniSonic: Towards Universal and Holistic Audio Generation from Video and Text
Weiguo Pian, Saksham Singh Kushwaha, Zhimin Chen +4
In this paper, we propose Universal Holistic Audio Generation (UniHAGen), a task for synthesizing comprehensive auditory scenes that include both on-screen and off-screen sounds ac…
Omni-MMSI: Toward Identity-attributed Social Interaction Understanding
Xinpeng Li, Bolin Lai, Hardy Chen +5
We introduce Omni-MMSI, a new task that requires comprehensive social interaction understanding from raw audio, vision, and speech input. The task involves perceiving identity-attr…
ARGaze: Autoregressive Transformers for Online Egocentric Gaze Estimation
Jia Li, Wenjie Zhao, Shijian Deng +6
Online egocentric gaze estimation predicts where a camera wearer is looking from first-person video using only past and current frames, a task essential for augmented reality and a…
Explainable AI-Generated Image Detection RewardBench
Michael Yang, Shijian Deng, William T. Doan +4
Conventional, classification-based AI-generated image detection methods cannot explain why an image is considered real or AI-generated in a way a human expert would, which reduces…
Toward Gaze Target Detection of Young Autistic Children
Shijian Deng, Erin E. Kosloski, Siva Sai Nagender Vasireddy +6
The automatic detection of gaze targets in autistic children through artificial intelligence can be impactful, especially for those who lack access to a sufficient number of profes…
Self-Improvement in Multimodal Large Language Models: A Survey
Shijian Deng, Kai Wang, Tianyu Yang +2
Recent advancements in self-improvement for Large Language Models (LLMs) have efficiently enhanced model capabilities without significantly increasing costs, particularly in terms…
Towards Online Multi-Modal Social Interaction Understanding
Xinpeng Li, Shijian Deng, Bolin Lai +3
In this paper, we introduce a new problem, Online-MMSI, where the model must perform multimodal social interaction understanding (MMSI) using only historical information. Given a r…
Modality-Inconsistent Continual Learning of Multimodal Large Language Models
Weiguo Pian, Shijian Deng, Shentong Mo +3
In this paper, we introduce Modality-Inconsistent Continual Learning (MICL), a new continual learning scenario for Multimodal Large Language Models (MLLMs) that involves tasks with…
Efficient Self-Improvement in Multimodal Large Language Models: A Model-Level Judge-Free Approach
Shijian Deng, Wentian Zhao, Yu-Jhe Li +4
Self-improvement in multimodal large language models (MLLMs) is crucial for enhancing their reliability and robustness. However, current methods often rely heavily on MLLMs themsel…
Continual Audio-Visual Sound Separation
Weiguo Pian, Yiyang Nan, Shijian Deng +3
In this paper, we introduce a novel continual audio-visual sound separation task, aiming to continuously separate sound sources for new classes while preserving performance on prev…
AV-DiT: Efficient Audio-Visual Diffusion Transformer for Joint Audio and Video Generation
Kai Wang, Shijian Deng, Jing Shi +2
Recent Diffusion Transformers (DiTs) have shown impressive capabilities in generating high-quality single-modality content, including images, videos, and audio. However, it is stil…
Hear Me, See Me, Understand Me: Audio-Visual Autism Behavior Recognition
Shijian Deng, Erin E. Kosloski, Siddhi Patel +9
In this article, we introduce a novel problem of audio-visual autism behavior recognition, which includes social behavior recognition, an essential aspect previously omitted in AI-…
Separating Invisible Sounds Toward Universal Audiovisual Scene-Aware Sound Separation
Yiyang Su, Ali Vosoughi, Shijian Deng +2
The audio-visual sound separation field assumes visible sources in videos, but this excludes invisible sounds beyond the camera's view. Current methods struggle with such sounds la…
Unveiling Cross Modality Bias in Visual Question Answering: A Causal View with Possible Worlds VQA
Ali Vosoughi, Shijian Deng, Songyang Zhang +3
To increase the generalization capability of VQA systems, many recent studies have tried to de-bias spurious language or vision associations that shortcut the question or image to…