1.2k citations · 1.8k across the 43 of their papers we have counts for
34 papers · 1 filter
Separating Invisible Sounds Toward Universal Audiovisual Scene-Aware Sound Separation
Yiyang Su, Ali Vosoughi, Shijian Deng +2
The audio-visual sound separation field assumes visible sources in videos, but this excludes invisible sounds beyond the camera's view. Current methods struggle with such sounds la…
DiffI2I: Efficient Diffusion Model for Image-to-Image Translation
Bin Xia, Yulun Zhang, Shiyin Wang +6
The Diffusion Model (DM) has emerged as the SOTA approach for image synthesis. However, the existing DM cannot perform well on some image-to-image translation (I2I) tasks. Differen…
SignDiff: Diffusion Model for American Sign Language Production
Sen Fang, Chunyu Sui, Yanghao Zhou +4
In this paper, we propose a dual-condition diffusion pre-training model named SignDiff that can generate human sign language speakers from a skeleton pose. SignDiff has a novel Fra…
Audio-Visual Class-Incremental Learning
Weiguo Pian, Shentong Mo, Yunhui Guo +1
In this paper, we introduce audio-visual class-incremental learning, a class-incremental learning scenario for audio-visual video recognition. We demonstrate that joint audio-visua…
High-Quality Visually-Guided Sound Separation from Diverse Categories
Chao Huang, Susan Liang, Yapeng Tian +2
We propose DAVIS, a Diffusion-based Audio-VIsual Separation framework that solves the audio-visual sound source separation task through generative learning. Existing methods typica…
Unveiling Cross Modality Bias in Visual Question Answering: A Causal View with Possible Worlds VQA
Ali Vosoughi, Shijian Deng, Songyang Zhang +3
To increase the generalization capability of VQA systems, many recent studies have tried to de-bias spurious language or vision associations that shortcut the question or image to…
DiffAVA: Personalized Text-to-Audio Generation with Visual Alignment
Shentong Mo, Jing Shi, Yapeng Tian
Text-to-audio (TTA) generation is a recent popular problem that aims to synthesize general audio given text descriptions. Previous methods utilized latent diffusion models to learn…
AV-SAM: Segment Anything Model Meets Audio-Visual Localization and Segmentation
Shentong Mo, Yapeng Tian
Segment Anything Model (SAM) has recently shown its powerful effectiveness in visual segmentation tasks. However, there is less exploration concerning how SAM works on audio-visual…
EgoVSR: Towards High-Quality Egocentric Video Super-Resolution
Yichen Chi, Junhao Gu, Jiamiao Zhang +2
Due to the limitations of capture devices and scenarios, egocentric videos frequently have low visual quality, mainly caused by high compression and severe motion blur. With the in…
Audio-Visual Grouping Network for Sound Localization from Mixtures
Shentong Mo, Yapeng Tian
Sound source localization is a typical and challenging task that predicts the location of sound sources in a video. Previous single-source methods mainly used the audio-visual asso…
Egocentric Audio-Visual Object Localization
Chao Huang, Yapeng Tian, Anurag Kumar +1
Humans naturally perceive surrounding scenes by unifying sound and sight in a first-person view. Likewise, machines are advanced to approach human intelligence by learning with mul…
DiffIR: Efficient Diffusion Model for Image Restoration
Bin Xia, Yulun Zhang, Shiyin Wang +5
Diffusion model (DM) has achieved SOTA performance by modeling the image synthesis process into a sequential application of a denoising network. However, different from image synth…
AV-NeRF: Learning Neural Fields for Real-World Audio-Visual Scene Synthesis
Susan Liang, Chao Huang, Yapeng Tian +2
Can machines recording an audio-visual scene produce realistic, matching audio-visual experiences at novel positions and novel view directions? We answer it by studying a new task…
Basic Binary Convolution Unit for Binarized Image Restoration Network
Bin Xia, Yulun Zhang, Yitong Wang +4
Lighter and faster image restoration (IR) models are crucial for the deployment on resource-limited devices. Binary neural network (BNN), one of the most promising model compressio…
Learning in Audio-visual Context: A Review, Analysis, and New Perspective
Yake Wei, Di Hu, Yapeng Tian +1
Sight and hearing are two senses that play a vital role in human communication and scene understanding. To mimic human perception ability, audio-visual learning, aimed at developin…
Meta-Learning based Degradation Representation for Blind Super-Resolution
Bin Xia, Yapeng Tian, Yulun Zhang +3
The most of CNN based super-resolution (SR) methods assume that the degradation is known (\eg, bicubic). These methods will suffer a severe performance drop when the degradation is…
Structured Sparsity Learning for Efficient Video Super-Resolution
Bin Xia, Jingwen He, Yulun Zhang +4
The high computational costs of video super-resolution (VSR) models hinder their deployment on resource-limited devices, (e.g., smartphones and drones). Existing VSR models contain…
Learning to Answer Questions in Dynamic Audio-Visual Scenarios
Guangyao Li, Yake Wei, Yapeng Tian +3
In this paper, we focus on the Audio-Visual Question Answering (AVQA) task, which aims to answer questions regarding different visual objects, sounds, and their associations in vid…
Learning Spatio-Temporal Downsampling for Effective Video Upscaling
Xiaoyu Xiang, Yapeng Tian, Vijay Rengarajan +3
Downsampling is one of the most basic image processing operations. Improper spatio-temporal downsampling applied on videos can cause aliasing issues such as moiré patterns in space…
STDAN: Deformable Attention Network for Space-Time Video Super-Resolution
Hai Wang, Xiaoyu Xiang, Yapeng Tian +2
The target of space-time video super-resolution (STVSR) is to increase the spatial-temporal resolution of low-resolution (LR) and low frame rate (LFR) videos. Recent approaches bas…
Coarse-to-Fine Embedded PatchMatch and Multi-Scale Dynamic Aggregation for Reference-based Super-Resolution
Bin Xia, Yapeng Tian, Yucheng Hang +3
Reference-based super-resolution (RefSR) has made significant progress in producing realistic textures using an external reference (Ref) image. However, existing RefSR methods obta…
Efficient Non-Local Contrastive Attention for Image Super-Resolution
Bin Xia, Yucheng Hang, Yapeng Tian +3
Non-Local Attention (NLA) brings significant improvement for Single Image Super-Resolution (SISR) by leveraging intrinsic feature correlation in natural images. However, NLA gives…
Space-Time Memory Network for Sounding Object Localization in Videos
Sizhe Li, Yapeng Tian, Chenliang Xu
Leveraging temporal synchronization and association within sight and sound is an essential step towards robust localization of sounding objects. To this end, we propose a space-tim…
Zooming SlowMo: An Efficient One-Stage Framework for Space-Time Video Super-Resolution
Xiaoyu Xiang, Yapeng Tian, Yulun Zhang +3
In this paper, we address the space-time video super-resolution, which aims at generating a high-resolution (HR) slow-motion video from a low-resolution (LR) and low frame rate (LF…