551 citations · 2.3k across the 76 of their papers we have counts for
78 papers
KeyRec: Bounded Visual Memory for Streaming and Long-Video Understanding
Zihan Chen, Xuejian Rong, Xiaojuan Wang +4
Vision-language models are increasingly used to understand long videos and continuous streams. However, dense visual tokens accumulate with video duration, making long-context infe…
StochasT: Learning with Stochastic Turn Depth for Visual Instruction Tuning
Yuan Qing, Chengzhi Mao, Boqing Gong
Large Vision-Language Models (LVLMs) rely extensively on Visual Instruction Tuning (VIT) to elicit their multimodal reasoning capabilities. However, we find a discrepancy: VIT ofte…
Video Timeline Modeling For News Story Understanding
Meng Liu, Mingda Zhang, Jialu Liu +5
In this paper, we present a novel problem, namely video timeline modeling. Our objective is to create a video-associated timeline from a set of videos related to a specific topic,…
Multi-modal Domain Adaptation for REG via Relation Transfer
Yifan Ding, Liqiang Wang, Boqing Gong
Domain adaptation, which aims to transfer knowledge between domains, has been well studied in many areas such as image classification and object detection. However, for multi-modal…
VideoGLUE: Video General Understanding Evaluation of Foundation Models
Liangzhe Yuan, Nitesh Bharadwaj Gundavarapu, Long Zhao +14
We evaluate the video understanding capabilities of existing foundation models (FMs) using a carefully designed experiment protocol consisting of three hallmark tasks (action recog…
Identity Encoder for Personalized Diffusion
Yu-Chuan Su, Kelvin C. K. Chan, Yandong Li +5
Many applications can benefit from personalized image generation models, including image enhancement, video conferences, just to name a few. Existing works achieved personalization…
Domain Generalization with Adversarial Intensity Attack for Medical Image Segmentation
Zheyuan Zhang, Bin Wang, Lanhong Yao +5
Most statistical learning algorithms rely on an over-simplified assumption, that is, the train and test data are independent and identically distributed. In real-world scenarios, h…
Taming Encoder for Zero Fine-tuning Image Customization with Text-to-Image Diffusion Models
Xuhui Jia, Yang Zhao, Kelvin C. K. Chan +6
This paper proposes a method for generating images of customized objects specified by users. The method is based on a general framework that bypasses the lengthy optimization requi…
Federated Learning of Shareable Bases for Personalization-Friendly Image Classification
Hong-You Chen, Jike Zhong, Mingda Zhang +5
Personalized federated learning (PFL) aims to harness the collective wisdom of clients' data while building personalized models tailored to individual clients' data distributions.…
Structured Video-Language Modeling with Temporal Grouping and Spatial Grounding
Yuanhao Xiong, Long Zhao, Boqing Gong +5
Existing video-language pre-training methods primarily focus on instance-level alignment between video clips and captions via global contrastive learning but neglect rich fine-grai…
Unified Visual Relationship Detection with Vision and Language Models
Long Zhao, Liangzhe Yuan, Boqing Gong +5
This work focuses on training a single visual relationship detector predicting over the union of label spaces from multiple datasets. Merging labels spanning different datasets cou…
On Calibrating Semantic Segmentation Models: Analyses and An Algorithm
Dongdong Wang, Boqing Gong, Liqiang Wang
We study the problem of semantic segmentation calibration. Lots of solutions have been proposed to approach model miscalibration of confidence in image classification. However, to…
LESS: Label-Efficient Semantic Segmentation for LiDAR Point Clouds
Minghua Liu, Yin Zhou, Charles R. Qi +3
Semantic segmentation of LiDAR point clouds is an important task in autonomous driving. However, training deep models via conventional supervised methods requires large datasets wh…
Open Long-Tailed Recognition in a Dynamic World
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan +3
Real world data often exhibits a long-tailed and open-ended (with unseen classes) distribution. A practical recognition system must balance between majority (head) and minority (ta…
medXGAN: Visual Explanations for Medical Classifiers through a Generative Latent Space
Amil Dravid, Florian Schiffers, Boqing Gong +1
Despite the surge of deep learning in the past decade, some users are skeptical to deploy these models in practice due to their black-box nature. Specifically, in the medical space…
Surrogate Gap Minimization Improves Sharpness-Aware Training
Juntang Zhuang, Boqing Gong, Liangzhe Yuan +6
The recently proposed Sharpness-Aware Minimization (SAM) improves generalization by minimizing a \textit{perturbed loss} defined as the maximum loss within a neighborhood in the pa…
Towards a Unified Foundation Model: Jointly Pre-Training Transformers on Unpaired Images and Text
Qing Li, Boqing Gong, Yin Cui +4
In this paper, we explore the possibility of building a unified foundation model that can be adapted to both vision-only and text-only tasks. Starting from BERT and ViT, we design…
Exploring Temporal Granularity in Self-Supervised Video Representation Learning
Rui Qian, Yeqing Li, Liangzhe Yuan +7
This work presents a self-supervised learning framework named TeG to explore Temporal Granularity in learning video representations. In TeG, we sample a long clip from a video and…
Contextualized Spatio-Temporal Contrastive Learning with Self-Supervision
Liangzhe Yuan, Rui Qian, Yin Cui +5
Modern self-supervised learning algorithms typically enforce persistency of instance representations across views. While being very effective on learning holistic image and video r…
Anti-Neuron Watermarking: Protecting Personal Data Against Unauthorized Neural Networks
Zihang Zou, Boqing Gong, Liqiang Wang
We study protecting a user's data (images in this work) against a learner's unauthorized use in training neural networks. It is especially challenging when the user's data is only…
Federated Multi-Target Domain Adaptation
Chun-Han Yao, Boqing Gong, Yin Cui +3
Federated learning methods enable us to train machine learning models on distributed user data while preserving its privacy. However, it is not always feasible to obtain high-quali…
On Model Calibration for Long-Tailed Object Detection and Instance Segmentation
Tai-Yu Pan, Cheng Zhang, Yandong Li +5
Vanilla models for object detection and instance segmentation suffer from the heavy bias toward detecting frequent objects in the long-tailed setting. Existing methods address this…
Bridging the Gap Between Object Detection and User Intent via Query-Modulation
Marco Fornoni, Chaochao Yan, Liangchen Luo +5
When interacting with objects through cameras, or pictures, users often have a specific intent. For example, they may want to perform a visual search. With most object detection mo…
Adversarially Adaptive Normalization for Single Domain Generalization
Xinjie Fan, Qifei Wang, Junjie Ke +3
Single domain generalization aims to learn a model that performs well on many unseen domains with only one domain data for training. Existing works focus on studying the adversaria…