466 citations · 2.6k across the 60 of their papers we have counts for
62 papers
Tuning Multi-mode Token-level Prompt Alignment across Modalities
Dongsheng Wang, Miaoge Li, Xinyang Liu +3
Advancements in prompt tuning of vision-language models have underscored their potential in enhancing open-world visual concept comprehension. However, prior works only primarily f…
Towards Debiasing Frame Length Bias in Text-Video Retrieval via Causal Intervention
Burak Satar, Hongyuan Zhu, Hanwang Zhang +1
Many studies focus on improving pretraining or developing new backbones in text-video retrieval. However, existing methods may suffer from the learning and inference bias issue, as…
Make the U in UDA Matter: Invariant Consistency Learning for Unsupervised Domain Adaptation
Zhongqi Yue, Hanwang Zhang, Qianru Sun
Domain Adaptation (DA) is always challenged by the spurious correlation between domain-invariant features (e.g., class identity) and domain-specific features (e.g., environment) th…
Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructions
Juncheng Li, Kaihang Pan, Zhiqi Ge +7
Recent advancements in Multimodal Large Language Models (MLLMs) have been utilizing Visual Prompt Generators (VPGs) to convert visual features into tokens that LLMs can recognize.…
An Overview of Challenges in Egocentric Text-Video Retrieval
Burak Satar, Hongyuan Zhu, Hanwang Zhang +1
Text-video retrieval contains various challenges, including biases coming from diverse sources. We highlight some of them supported by illustrations to open a discussion. Besides,…
Fast Diffusion Model
Zike Wu, Pan Zhou, Kenji Kawaguchi +1
Diffusion models (DMs) have been adopted across diverse fields with its remarkable abilities in capturing intricate data distributions. In this paper, we propose a Fast Diffusion M…
Risk-Aware Objective-Based Forecasting in Inertia Management
Haipeng Zhang, Ran Li, Yan Chen +3
The objective-based forecasting considers the asymmetric and non-linear impacts of forecasting errors on decision objectives, thus improving the effectiveness of its downstream dec…
Learning Trajectory-Word Alignments for Video-Language Tasks
Xu Yang, Zhangzikang Li, Haiyang Xu +7
In a video, an object usually appears as the trajectory, i.e., it spans over a few spatial but longer temporal patches, that contains abundant spatiotemporal contexts. However, mod…
Learning to Collocate Visual-Linguistic Neural Modules for Image Captioning
Xu Yang, Hanwang Zhang, Chongyang Gao +1
Humans tend to decompose a sentence into different parts like \textsc{sth do sth at someplace} and then fill each part with certain content. Inspired by this, we follow the \textit…
RoME: Role-aware Mixture-of-Expert Transformer for Text-to-Video Retrieval
Burak Satar, Hongyuan Zhu, Hanwang Zhang +1
Seas of videos are uploaded daily with the popularity of social channels; thus, retrieving the most related video contents with user textual queries plays a more crucial role. Most…
Exploiting Semantic Role Contextualized Video Features for Multi-Instance Text-Video Retrieval EPIC-KITCHENS-100 Multi-Instance Retrieval Challenge 2022
Burak Satar, Hongyuan Zhu, Hanwang Zhang +1
In this report, we present our approach for EPIC-KITCHENS-100 Multi-Instance Retrieval Challenge 2022. We first parse sentences into semantic roles corresponding to verbs and nouns…
Distilling Causal Effect of Data in Class-Incremental Learning
Xinting Hu, Kaihua Tang, Chunyan Miao +2
We propose a causal framework to explain the catastrophic forgetting in Class-Incremental Learning (CIL) and then derive a novel distillation method that is orthogonal to the exist…
Causal Attention for Vision-Language Tasks
Xu Yang, Hanwang Zhang, Guojun Qi +1
We present a novel attention mechanism: Causal Attention (CATT), to remove the ever-elusive confounding effect in existing attention-based vision-language models. This effect cause…
DMotion: Robotic Visuomotor Control with Unsupervised Forward Model Learned from Videos
Haoqi Yuan, Ruihai Wu, Andrew Zhao +3
Learning an accurate model of the environment is essential for model-based control tasks. Existing methods in robotic visuomotor control usually learn from data with heavily labell…
Counterfactual Zero-Shot and Open-Set Visual Recognition
Zhongqi Yue, Tan Wang, Hanwang Zhang +2
We present a novel counterfactual framework for both Zero-Shot Learning (ZSL) and Open-Set Recognition (OSR), whose common challenge is generalizing to the unseen-classes by only t…
Sill-Net: Feature Augmentation with Separated Illumination Representation
Haipeng Zhang, Zhong Cao, Ziang Yan +1
For visual object recognition tasks, the illumination variations can cause distinct changes in object appearance and thus confuse the deep neural network based recognition models.…
Causal Intervention for Weakly-Supervised Semantic Segmentation
Dong Zhang, Hanwang Zhang, Jinhui Tang +2
We present a causal inference framework to improve Weakly-Supervised Semantic Segmentation (WSSS). Specifically, we aim to generate better pixel-level pseudo-masks by using only im…
Interventional Few-Shot Learning
Zhongqi Yue, Hanwang Zhang, Qianru Sun +1
We uncover an ever-overlooked deficiency in the prevailing Few-Shot Learning (FSL) methods: the pre-trained knowledge is indeed a confounder that limits the performance. This findi…
Long-Tailed Classification by Keeping the Good and Removing the Bad Momentum Causal Effect
Kaihua Tang, Jianqiang Huang, Hanwang Zhang
As the class size grows, maintaining a balanced dataset across many classes is challenging because the data are long-tailed in nature; it is even impossible when the sample-of-inte…
Ref-NMS: Breaking Proposal Bottlenecks in Two-Stage Referring Expression Grounding
Long Chen, Wenbo Ma, Jun Xiao +2
The prevailing framework for solving referring expression grounding is based on a two-stage process: 1) detecting proposals with an object detector and 2) grounding the referent to…
Feature Pyramid Transformer
Dong Zhang, Hanwang Zhang, Jinhui Tang +3
Feature interactions across space and scales underpin modern visual recognition systems because they introduce beneficial visual contexts. Conventionally, spatial contexts are pass…
Counterfactual VQA: A Cause-Effect Look at Language Bias
Yulei Niu, Kaihua Tang, Hanwang Zhang +3
VQA models may tend to rely on language bias as a shortcut and thus fail to sufficiently learn the multi-modal knowledge from both vision and language. Recent debiasing methods pro…
Iterative Context-Aware Graph Inference for Visual Dialog
Dan Guo, Hui Wang, Hanwang Zhang +2
Visual dialog is a challenging task that requires the comprehension of the semantic dependencies among implicit visual and textual contexts. This task can refer to the relation inf…
Learning to Segment the Tail
Xinting Hu, Yi Jiang, Kaihua Tang +3
Real-world visual recognition requires handling the extreme sample imbalance in large-scale long-tailed data. We propose a "divide&conquer" strategy for the challenging LVIS task:…