32 citations · 72 across the 14 of their papers we have counts for
14 papers
Progressive Spatio-temporal Perception for Audio-Visual Question Answering
Guangyao Li, Wenxuan Hou, Di Hu
Audio-Visual Question Answering (AVQA) task aims to answer questions about different visual objects, sounds, and their associations in videos. Such naturally multi-modal videos are…
Towards Long Form Audio-visual Video Understanding
Wenxuan Hou, Guangyao Li, Yapeng Tian +1
We live in a world filled with never-ending streams of multimodal information. As a more natural recording of the real scenario, long form audio-visual videos are expected as an im…
Multi-Scale Attention for Audio Question Answering
Guangyao Li, Yixin Xu, Di Hu
Audio question answering (AQA), acting as a widely used proxy task to explore scene understanding, has got more attention. The AQA is challenging for it requires comprehensive temp…
MMCosine: Multi-Modal Cosine Loss Towards Balanced Audio-Visual Fine-Grained Learning
Ruize Xu, Ruoxuan Feng, Shi-Xiong Zhang +1
Audio-visual learning helps to comprehensively understand the world by fusing practical information from multiple modalities. However, recent studies show that the imbalanced optim…
Revisiting Pre-training in Audio-Visual Learning
Ruoxuan Feng, Wenke Xia, Di Hu
Pre-training technique has gained tremendous success in enhancing model performance on various tasks, but found to perform worse than training from scratch in some uni-modal situat…
Balanced Audiovisual Dataset for Imbalance Analysis
Wenke Xia, Xu Zhao, Xincheng Pang +2
The imbalance problem is widespread in the field of machine learning, which also exists in multimodal learning areas caused by the intrinsic discrepancy between modalities of sampl…
TikTalk: A Video-Based Dialogue Dataset for Multi-Modal Chitchat in Real World
Hongpeng Lin, Ludan Ruan, Wenke Xia +8
To facilitate the research on intelligent and human-like chatbots with multi-modal context, we introduce a new video-based multi-modal dialogue dataset, called TikTalk. We collect…
Learning in Audio-visual Context: A Review, Analysis, and New Perspective
Yake Wei, Di Hu, Yapeng Tian +1
Sight and hearing are two senses that play a vital role in human communication and scene understanding. To mimic human perception ability, audio-visual learning, aimed at developin…
Dual Domain-Adversarial Learning for Audio-Visual Saliency Prediction
Yingzi Fan, Longfei Han, Yue Zhang +3
Both visual and auditory information are valuable to determine the salient regions in videos. Deep convolution neural networks (CNN) showcase strong capacity in coping with the aud…
Learning to Answer Questions in Dynamic Audio-Visual Scenarios
Guangyao Li, Yake Wei, Yapeng Tian +3
In this paper, we focus on the Audio-Visual Question Answering (AVQA) task, which aims to answer questions regarding different visual objects, sounds, and their associations in vid…
Balanced Multimodal Learning via On-the-fly Gradient Modulation
Xiaokang Peng, Yake Wei, Andong Deng +2
Multimodal learning helps to comprehensively understand the world, by integrating different senses. Accordingly, multiple input modalities are expected to boost model performance,…
SeCo: Separating Unknown Musical Visual Sounds with Consistency Guidance
Xinchi Zhou, Dongzhan Zhou, Wanli Ouyang +3
Recent years have witnessed the success of deep learning on the visual sound separation task. However, existing works follow similar settings where the training and testing dataset…
Visual Sound Localization in the Wild by Cross-Modal Interference Erasing
Xian Liu, Rui Qian, Hang Zhou +5
The task of audio-visual sound source localization has been well studied under constrained scenes, where the audio recordings are clean. However, in real-world scenarios, audios ar…
Class-aware Sounding Objects Localization via Audiovisual Correspondence
Di Hu, Yake Wei, Rui Qian +3
Audiovisual scenes are pervasive in our daily life. It is commonplace for humans to discriminatively localize different sounding objects but quite challenging for machines to achie…