412 citations · 1.1k across the 29 of their papers we have counts for
32 papers
Textual Response-Aware Guidance for Zero-Shot Adversarial Robustness of Vision-Language Models
Yubo Cui, Xianchao Guan, Zheng Zhang
Vision-language models (VLMs) exhibit strong zero-shot generalization but remain vulnerable to adversarial attacks. Existing adversarial fine-tuning methods often use pre-trained m…
AGFT: Alignment-Guided Fine-Tuning for Zero-Shot Adversarial Robustness of Vision-Language Models
Yubo Cui, Xianchao Guan, Zijun Xiong +1
Pre-trained vision-language models (VLMs) exhibit strong zero-shot generalization but remain vulnerable to adversarial perturbations. Existing classification-guided adversarial fin…
Rényi Entropy: A New Token Pruning Metric for Vision Transformers
Wei-Yuan Su, Ruijie Zhang, Zheng Zhang
Vision Transformers (ViTs) achieve state-of-the-art performance but suffer from the complexity of self-attention, making inference costly for high-resolution inputs. To ad…
DETR Doesn't Need Multi-Scale or Locality Design
Yutong Lin, Yuhui Yuan, Zheng Zhang +3
This paper presents an improved DETR detector that maintains a "plain" nature: using a single-scale feature map and global cross-attention calculations without specific locality co…
DeepMIM: Deep Supervision for Masked Image Modeling
Sucheng Ren, Fangyun Wei, Samuel Albanie +2
Deep supervision, which involves extra supervisions to the intermediate features of a neural network, was widely used in image classification in the early deep learning era since i…
All in Tokens: Unifying Output Space of Visual Tasks via Soft Token
Jia Ning, Chen Li, Zheng Zhang +4
Unlike language tasks, where the output space is usually limited to a set of tokens, the output space of visual tasks is more complicated, making it difficult to build a unified vi…
TinyMIM: An Empirical Study of Distilling MIM Pre-trained Models
Sucheng Ren, Fangyun Wei, Zheng Zhang +1
Masked image modeling (MIM) performs strongly in pre-training large vision Transformers (ViTs). However, small models that are critical for real-world applications cannot or only m…
Exploring Discrete Diffusion Models for Image Captioning
Zixin Zhu, Yixuan Wei, Jianfeng Wang +7
The image captioning task is typically realized by an auto-regressive method that decodes the text tokens one by one. We present a diffusion-based captioning model, dubbed the name…
Could Giant Pretrained Image Models Extract Universal Representations?
Yutong Lin, Ze Liu, Zheng Zhang +4
Frozen pretrained models have become a viable alternative to the pretraining-then-finetuning paradigm for transfer learning. However, with frozen models there are relatively few pa…
Expediting Large-Scale Vision Transformer for Dense Prediction without Fine-tuning
Weicong Liang, Yuhui Yuan, Henghui Ding +6
Vision transformers have recently achieved competitive results across various vision tasks but still suffer from heavy computation costs when processing a large number of tokens. M…
On Data Scaling in Masked Image Modeling
Zhenda Xie, Zheng Zhang, Yue Cao +4
An important goal of self-supervised learning is to enable model pre-training to benefit from almost unlimited data. However, one method that has recently become popular, namely ma…
Revealing the Dark Secrets of Masked Image Modeling
Zhenda Xie, Zigang Geng, Jingcheng Hu +3
Masked image modeling (MIM) as pre-training is shown to be effective for numerous vision downstream tasks, but how and where MIM works remain unclear. In this paper, we compare MIM…
Contrastive Learning Rivals Masked Image Modeling in Fine-tuning via Feature Distillation
Yixuan Wei, Han Hu, Zhenda Xie +5
Masked image modeling (MIM) learns representations with remarkably good fine-tuning performances, overshadowing previous prevalent pre-training approaches such as image classificat…
iCAR: Bridging Image Classification and Image-text Alignment for Visual Recognition
Yixuan Wei, Yue Cao, Zheng Zhang +4
Image classification, which classifies images by pre-defined categories, has been the dominant approach to visual representation learning over the last decade. Visual learning thro…
A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Model
Mengde Xu, Zheng Zhang, Fangyun Wei +4
Recently, open-vocabulary image classification by vision language pre-training has demonstrated incredible achievements, that the model can classify arbitrary categories without se…
Bootstrap Your Object Detector via Mixed Training
Mengde Xu, Zheng Zhang, Fangyun Wei +5
We introduce MixTraining, a new training paradigm for object detection that can improve the performance of existing detectors for free. MixTraining enhances data augmentation by ut…
SimMIM: A Simple Framework for Masked Image Modeling
Zhenda Xie, Zheng Zhang, Yue Cao +5
This paper presents SimMIM, a simple framework for masked image modeling. We simplify recently proposed related approaches without special designs such as block-wise masking and to…
Swin Transformer V2: Scaling Up Capacity and Resolution
Ze Liu, Han Hu, Yutong Lin +9
Large-scale NLP models have been shown to significantly improve the performance on language tasks with no signs of saturation. They also demonstrate amazing few-shot capabilities l…
Video Swin Transformer
Ze Liu, Jia Ning, Yue Cao +4
The vision community is witnessing a modeling shift from CNNs to Transformers, where pure Transformer architectures have attained top accuracy on the major video recognition benchm…
End-to-End Semi-Supervised Object Detection with Soft Teacher
Mengde Xu, Zheng Zhang, Han Hu +5
This paper presents an end-to-end semi-supervised object detection approach, in contrast to previous more complex multi-stage methods. The end-to-end training gradually improves ps…
Self-Supervised Learning with Swin Transformers
Zhenda Xie, Yutong Lin, Zhuliang Yao +4
We are witnessing a modeling shift from CNN to Transformers in computer vision. In this work, we present a self-supervised learning approach called MoBY, with Vision Transformers a…
Breaking Shortcut: Exploring Fully Convolutional Cycle-Consistency for Video Correspondence Learning
Yansong Tang, Zhenyu Jiang, Zhenda Xie +4
Previous cycle-consistency correspondence learning methods usually leverage image patches for training. In this paper, we present a fully convolutional method, which is simpler and…
Group-Free 3D Object Detection via Transformers
Ze Liu, Zheng Zhang, Yue Cao +2
Recently, directly detecting 3D objects from 3D point clouds has received increasing attention. To extract object representation from an irregular point cloud, existing methods usu…
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Ze Liu, Yutong Lin, Yue Cao +5
This paper presents a new vision Transformer, called Swin Transformer, that capably serves as a general-purpose backbone for computer vision. Challenges in adapting Transformer fro…