941 citations · 5.4k across the 87 of their papers we have counts for
7 papers · 2 filters
A Unified View of Masked Image Modeling
Zhiliang Peng, Li Dong, Hangbo Bao +2
Masked image modeling has demonstrated great potential to eliminate the label-hungry problem of training large-scale vision Transformers, achieving impressive performance on variou…
Non-Contrastive Learning Meets Language-Image Pre-Training
Jinghao Zhou, Li Dong, Zhe Gan +2
Contrastive language-image pre-training (CLIP) serves as a de-facto standard to align images and texts. Nonetheless, the loose correlation between images and texts of web-crawled d…
Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks
Wenhui Wang, Hangbo Bao, Li Dong +8
A big convergence of language, vision, and multimodal pretraining is emerging. In this work, we introduce a general-purpose multimodal foundation model BEiT-3, which achieves state…
BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizers
Zhiliang Peng, Li Dong, Hangbo Bao +2
Masked image modeling (MIM) has demonstrated impressive results in self-supervised representation learning by recovering corrupted image patches. However, most existing studies ope…
VL-BEiT: Generative Vision-Language Pretraining
Hangbo Bao, Wenhui Wang, Li Dong +1
We introduce a vision-language foundation model called VL-BEiT, which is a bidirectional multimodal Transformer learned by generative pretraining. Our minimalist solution conducts…
CLIP Models are Few-shot Learners: Empirical Studies on VQA and Visual Entailment
Haoyu Song, Li Dong, Wei-Nan Zhang +2
CLIP has shown a remarkable zero-shot capability on a wide range of vision tasks. Previously, CLIP is only regarded as a powerful visual encoder. However, after being pre-trained b…
Corrupted Image Modeling for Self-Supervised Visual Pre-Training
Yuxin Fang, Li Dong, Hangbo Bao +2
We introduce Corrupted Image Modeling (CIM) for self-supervised visual pre-training. CIM uses an auxiliary generator with a small trainable BEiT to corrupt the input image instead…