1.2k citations · 1.8k across the 43 of their papers we have counts for
43 papers
LAVSS: Location-Guided Audio-Visual Spatial Audio Separation
Yuxin Ye, Wenming Yang, Yapeng Tian
Existing machine learning research has achieved promising results in monaural audio-visual separation (MAVS). However, most MAVS methods purely consider what the sound source is, n…
Separating Invisible Sounds Toward Universal Audiovisual Scene-Aware Sound Separation
Yiyang Su, Ali Vosoughi, Shijian Deng +2
The audio-visual sound separation field assumes visible sources in videos, but this excludes invisible sounds beyond the camera's view. Current methods struggle with such sounds la…
Neural Acoustic Context Field: Rendering Realistic Room Impulse Response With Neural Fields
Susan Liang, Chao Huang, Yapeng Tian +2
Room impulse response (RIR), which measures the sound propagation within an environment, is critical for synthesizing high-fidelity audio for a given environment. Some prior work h…
DiffI2I: Efficient Diffusion Model for Image-to-Image Translation
Bin Xia, Yulun Zhang, Shiyin Wang +6
The Diffusion Model (DM) has emerged as the SOTA approach for image synthesis. However, the existing DM cannot perform well on some image-to-image translation (I2I) tasks. Differen…
SignDiff: Diffusion Model for American Sign Language Production
Sen Fang, Chunyu Sui, Yanghao Zhou +4
In this paper, we propose a dual-condition diffusion pre-training model named SignDiff that can generate human sign language speakers from a skeleton pose. SignDiff has a novel Fra…
Audio-Visual Class-Incremental Learning
Weiguo Pian, Shentong Mo, Yunhui Guo +1
In this paper, we introduce audio-visual class-incremental learning, a class-incremental learning scenario for audio-visual video recognition. We demonstrate that joint audio-visua…
High-Quality Visually-Guided Sound Separation from Diverse Categories
Chao Huang, Susan Liang, Yapeng Tian +2
We propose DAVIS, a Diffusion-based Audio-VIsual Separation framework that solves the audio-visual sound source separation task through generative learning. Existing methods typica…
PEANUT: A Human-AI Collaborative Tool for Annotating Audio-Visual Data
Zheng Zhang, Zheng Ning, Chenliang Xu +2
Audio-visual learning seeks to enhance the computer's multi-modal perception leveraging the correlation between the auditory and visual modalities. Despite their many useful downst…
Dual Arbitrary Scale Super-Resolution for Multi-Contrast MRI
Jiamiao Zhang, Yichen Chi, Jun Lyu +2
Limited by imaging systems, the reconstruction of Magnetic Resonance Imaging (MRI) images from partial measurement is essential to medical imaging research. Benefiting from the div…
Towards Long Form Audio-visual Video Understanding
Wenxuan Hou, Guangyao Li, Yapeng Tian +1
We live in a world filled with never-ending streams of multimodal information. As a more natural recording of the real scenario, long form audio-visual videos are expected as an im…
Unveiling Cross Modality Bias in Visual Question Answering: A Causal View with Possible Worlds VQA
Ali Vosoughi, Shijian Deng, Songyang Zhang +3
To increase the generalization capability of VQA systems, many recent studies have tried to de-bias spurious language or vision associations that shortcut the question or image to…
DiffAVA: Personalized Text-to-Audio Generation with Visual Alignment
Shentong Mo, Jing Shi, Yapeng Tian
Text-to-audio (TTA) generation is a recent popular problem that aims to synthesize general audio given text descriptions. Previous methods utilized latent diffusion models to learn…
AV-SAM: Segment Anything Model Meets Audio-Visual Localization and Segmentation
Shentong Mo, Yapeng Tian
Segment Anything Model (SAM) has recently shown its powerful effectiveness in visual segmentation tasks. However, there is less exploration concerning how SAM works on audio-visual…
EgoVSR: Towards High-Quality Egocentric Video Super-Resolution
Yichen Chi, Junhao Gu, Jiamiao Zhang +2
Due to the limitations of capture devices and scenarios, egocentric videos frequently have low visual quality, mainly caused by high compression and severe motion blur. With the in…
Audio-Visual Grouping Network for Sound Localization from Mixtures
Shentong Mo, Yapeng Tian
Sound source localization is a typical and challenging task that predicts the location of sound sources in a video. Previous single-source methods mainly used the audio-visual asso…
Egocentric Audio-Visual Object Localization
Chao Huang, Yapeng Tian, Anurag Kumar +1
Humans naturally perceive surrounding scenes by unifying sound and sight in a first-person view. Likewise, machines are advanced to approach human intelligence by learning with mul…
DiffIR: Efficient Diffusion Model for Image Restoration
Bin Xia, Yulun Zhang, Shiyin Wang +5
Diffusion model (DM) has achieved SOTA performance by modeling the image synthesis process into a sequential application of a denoising network. However, different from image synth…
AV-NeRF: Learning Neural Fields for Real-World Audio-Visual Scene Synthesis
Susan Liang, Chao Huang, Yapeng Tian +2
Can machines recording an audio-visual scene produce realistic, matching audio-visual experiences at novel positions and novel view directions? We answer it by studying a new task…
Knowledge Distillation based Degradation Estimation for Blind Super-Resolution
Bin Xia, Yulun Zhang, Yitong Wang +4
Blind image super-resolution (Blind-SR) aims to recover a high-resolution (HR) image from its corresponding low-resolution (LR) input image with unknown degradations. Most of the e…
Basic Binary Convolution Unit for Binarized Image Restoration Network
Bin Xia, Yulun Zhang, Yitong Wang +4
Lighter and faster image restoration (IR) models are crucial for the deployment on resource-limited devices. Binary neural network (BNN), one of the most promising model compressio…
Learning in Audio-visual Context: A Review, Analysis, and New Perspective
Yake Wei, Di Hu, Yapeng Tian +1
Sight and hearing are two senses that play a vital role in human communication and scene understanding. To mimic human perception ability, audio-visual learning, aimed at developin…
Meta-Learning based Degradation Representation for Blind Super-Resolution
Bin Xia, Yapeng Tian, Yulun Zhang +3
The most of CNN based super-resolution (SR) methods assume that the degradation is known (\eg, bicubic). These methods will suffer a severe performance drop when the degradation is…
Structured Sparsity Learning for Efficient Video Super-Resolution
Bin Xia, Jingwen He, Yulun Zhang +4
The high computational costs of video super-resolution (VSR) models hinder their deployment on resource-limited devices, (e.g., smartphones and drones). Existing VSR models contain…
Learning to Answer Questions in Dynamic Audio-Visual Scenarios
Guangyao Li, Yake Wei, Yapeng Tian +3
In this paper, we focus on the Audio-Visual Question Answering (AVQA) task, which aims to answer questions regarding different visual objects, sounds, and their associations in vid…