216 citations · 802 across the 61 of their papers we have counts for
61 papers
High-Quality Visually-Guided Sound Separation from Diverse Categories
Chao Huang, Susan Liang, Yapeng Tian +2
We propose DAVIS, a Diffusion-based Audio-VIsual Separation framework that solves the audio-visual sound source separation task through generative learning. Existing methods typica…
PEANUT: A Human-AI Collaborative Tool for Annotating Audio-Visual Data
Zheng Zhang, Zheng Ning, Chenliang Xu +2
Audio-visual learning seeks to enhance the computer's multi-modal perception leveraging the correlation between the auditory and visual modalities. Despite their many useful downst…
Unveiling Cross Modality Bias in Visual Question Answering: A Causal View with Possible Worlds VQA
Ali Vosoughi, Shijian Deng, Songyang Zhang +3
To increase the generalization capability of VQA systems, many recent studies have tried to de-bias spurious language or vision associations that shortcut the question or image to…
One Forward is Enough for Neural Network Training via Likelihood Ratio Method
Jinyang Jiang, Zeliang Zhang, Chenliang Xu +2
While backpropagation (BP) is the mainstream approach for gradient computation in neural network training, its heavy reliance on the chain rule of differentiation constrains the de…
Egocentric Audio-Visual Object Localization
Chao Huang, Yapeng Tian, Anurag Kumar +1
Humans naturally perceive surrounding scenes by unifying sound and sight in a first-person view. Likewise, machines are advanced to approach human intelligence by learning with mul…
AV-NeRF: Learning Neural Fields for Real-World Audio-Visual Scene Synthesis
Susan Liang, Chao Huang, Yapeng Tian +2
Can machines recording an audio-visual scene produce realistic, matching audio-visual experiences at novel positions and novel view directions? We answer it by studying a new task…
Improving Adversarial Transferability with Scheduled Step Size and Dual Example
Zeliang Zhang, Peihan Liu, Xiaosen Wang +1
Deep neural networks are widely known to be vulnerable to adversarial examples, especially showing significantly poor performance on adversarial examples generated under the white-…
A Whac-A-Mole Dilemma: Shortcuts Come in Multiples Where Mitigating One Amplifies Others
Zhiheng Li, Ivan Evtimov, Albert Gordo +5
Machine learning models have been found to learn shortcuts -- unintended decision rules that are unable to generalize -- undermining models' reliability. Previous works address thi…
Discover and Mitigate Unknown Biases with Debiasing Alternate Networks
Zhiheng Li, Anthony Hoogs, Chenliang Xu
Deep image classifiers have been found to learn biases from datasets. To mitigate the biases, most previous methods require labels of protected attributes (e.g., age, skin tone) as…
Learning to Answer Questions in Dynamic Audio-Visual Scenarios
Guangyao Li, Yake Wei, Yapeng Tian +3
In this paper, we focus on the Audio-Visual Question Answering (AVQA) task, which aims to answer questions regarding different visual objects, sounds, and their associations in vid…
StyleT2I: Toward Compositional and High-Fidelity Text-to-Image Synthesis
Zhiheng Li, Martin Renqiang Min, Kai Li +1
Although progress has been made for text-to-image synthesis, previous methods fall short of generalizing to unseen or underrepresented attribute compositions in the input text. Lac…
Transformer-empowered Multi-scale Contextual Matching and Aggregation for Multi-contrast MRI Super-resolution
Guangyuan Li, Jun Lv, Yapeng Tian +4
Magnetic resonance imaging (MRI) can present multi-contrast images of the same anatomical structures, enabling multi-contrast super-resolution (SR) techniques. Compared with SR rec…
Cross-modal Contrastive Distillation for Instructional Activity Anticipation
Zhengyuan Yang, Jingen Liu, Jing Huang +4
In this study, we aim to predict the plausible future action steps given an observation of the past and study the task of instructional activity anticipation. Unlike previous antic…
Anomaly Crossing: New Horizons for Video Anomaly Detection as Cross-domain Few-shot Learning
Guangyu Sun, Zhang Liu, Lianggong Wen +2
Video anomaly detection aims to identify abnormal events that occurred in videos. Since anomalous events are relatively rare, it is not feasible to collect a balanced dataset and t…
SpaceEdit: Learning a Unified Editing Space for Open-Domain Image Editing
Jing Shi, Ning Xu, Haitian Zheng +3
Recently, large pretrained models (e.g., BERT, StyleGAN, CLIP) have shown great knowledge transfer and generalization capability on various downstream tasks within their domains. I…
Space-Time Memory Network for Sounding Object Localization in Videos
Sizhe Li, Yapeng Tian, Chenliang Xu
Leveraging temporal synchronization and association within sight and sound is an essential step towards robust localization of sounding objects. To this end, we propose a space-tim…
Procedure Planning in Instructional Videos via Contextual Modeling and Model-based Policy Learning
Jing Bi, Jiebo Luo, Chenliang Xu
Learning new skills by observing humans' behaviors is an essential capability of AI. In this work, we leverage instructional videos to study humans' decision-making processes, focu…
Learning to Generate Scene Graph from Natural Language Supervision
Yiwu Zhong, Jing Shi, Jianwei Yang +2
Learning from image-text data has demonstrated recent success for many recognition tasks, yet is currently limited to visual features or individual visual concepts such as objects.…
Learning by Planning: Language-Guided Global Image Editing
Jing Shi, Ning Xu, Yihang Xu +3
Recently, language-guided global image editing draws increasing attention with growing application potentials. However, previous GAN-based methods are not only confined to domain-s…
Zooming SlowMo: An Efficient One-Stage Framework for Space-Time Video Super-Resolution
Xiaoyu Xiang, Yapeng Tian, Yulun Zhang +3
In this paper, we address the space-time video super-resolution, which aims at generating a high-resolution (HR) slow-motion video from a low-resolution (LR) and low frame rate (LF…
Cyclic Co-Learning of Sounding Object Visual Grounding and Sound Separation
Yapeng Tian, Di Hu, Chenliang Xu
There are rich synchronized audio and visual events in our daily life. Inside the events, audio scenes are associated with the corresponding visual objects; meanwhile, sounding obj…
Can audio-visual integration strengthen robustness under multimodal attacks?
Yapeng Tian, Chenliang Xu
In this paper, we propose to make a systematic study on machines multisensory perception under attacks. We use the audio-visual event recognition task against multimodal adversaria…
Discover the Unknown Biased Attribute of an Image Classifier
Zhiheng Li, Chenliang Xu
Recent works find that AI algorithms learn biases from data. Therefore, it is urgent and vital to identify biases in AI algorithms. However, the previous bias identification pipeli…
High-fidelity Face Tracking for AR/VR via Deep Lighting Adaptation
Lele Chen, Chen Cao, Fernando De la Torre +3
3D video avatars can empower virtual communications by providing compression, privacy, entertainment, and a sense of presence in AR/VR. Best 3D photo-realistic AR/VR avatars driven…