342 citations · 1.4k across the 24 of their papers we have counts for
26 papers
Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone
Zi-Yi Dou, Aishwarya Kamath, Zhe Gan +9
Vision-language (VL) pre-training has recently received considerable attention. However, most existing end-to-end pre-training approaches either only aim to tackle VL tasks such as…
GLIPv2: Unifying Localization and Vision-Language Understanding
Haotian Zhang, Pengchuan Zhang, Xiaowei Hu +7
We present GLIPv2, a grounded VL understanding model, that serves both localization tasks (e.g., object detection, instance segmentation) and Vision-Language (VL) understanding tas…
Detection Hub: Unifying Object Detection Datasets via Query Adaptation on Language Embedding
Lingchen Meng, Xiyang Dai, Yinpeng Chen +7
Combining multiple datasets enables performance boost on many computer vision tasks. But similar trend has not been witnessed in object detection when combining multiple datasets d…
Unified Contrastive Learning in Image-Text-Label Space
Jianwei Yang, Chunyuan Li, Pengchuan Zhang +4
Visual recognition is recently learned via either supervised learning on human-annotated image-label data or language-image contrastive learning with webly-crawled image-text pairs…
K-LITE: Learning Transferable Visual Models with External Knowledge
Sheng Shen, Chunyuan Li, Xiaowei Hu +11
The new generation of state-of-the-art computer vision systems are trained from natural language supervision, ranging from simple object category names to descriptive captions. Thi…
ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models
Chunyuan Li, Haotian Liu, Liunian Harold Li +8
Learning visual representations from natural language supervision has recently shown great promise in a number of pioneering works. In general, these language-augmented visual mode…
Vision-Language Intelligence: Tasks, Representation Learning, and Large Models
Feng Li, Hao Zhang, Yi-Fan Zhang +5
This paper presents a comprehensive survey of vision-language (VL) intelligence from the perspective of time. This survey is inspired by the remarkable progress in both computer vi…
Parameter-efficient Model Adaptation for Vision Transformers
Xuehai He, Chunyuan Li, Pengchuan Zhang +2
In computer vision, it has achieved great transfer learning performance via adapting large-scale pretrained vision models (e.g., vision transformers) to downstream tasks. Common ap…
RegionCLIP: Region-based Language-Image Pretraining
Yiwu Zhong, Jianwei Yang, Pengchuan Zhang +8
Contrastive language-image pretraining (CLIP) using image-text pairs has achieved impressive results on image classification in both zero-shot and transfer learning settings. Howev…
Grounded Language-Image Pre-training
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang +9
This paper presents a grounded language-image pre-training (GLIP) model for learning object-level, language-aware, and semantic-rich visual representations. GLIP unifies object det…
Florence: A New Foundation Model for Computer Vision
Lu Yuan, Dongdong Chen, Yi-Ling Chen +20
Automated visual understanding of our diverse and open world demands computer vision models to generalize well with minimal customization for specific tasks, similar to human visio…
An Empirical Study of Training End-to-End Vision-and-Language Transformers
Zi-Yi Dou, Yichong Xu, Zhe Gan +9
Vision-and-language (VL) pre-training has proven to be highly effective on various VL downstream tasks. While recent work has shown that fully transformer-based VL models can be mo…
Image Scene Graph Generation (SGG) Benchmark
Xiaotian Han, Jianwei Yang, Houdong Hu +3
There is a surge of interest in image scene graph generation (object, attribute and relationship detection) due to the need of building fine-grained image understanding models that…
Focal Self-attention for Local-Global Interactions in Vision Transformers
Jianwei Yang, Chunyuan Li, Pengchuan Zhang +4
Recently, Vision Transformer and its variants have shown great promise on various computer vision tasks. The ability of capturing short- and long-range visual dependencies through…
Efficient Self-supervised Vision Transformers for Representation Learning
Chunyuan Li, Jianwei Yang, Pengchuan Zhang +5
This paper investigates two techniques for developing efficient self-supervised vision transformers (EsViT) for visual representation learning. First, we show through a comprehensi…
Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encoding
Pengchuan Zhang, Xiyang Dai, Jianwei Yang +4
This paper presents a new Vision Transformer (ViT) architecture Multi-Scale Vision Longformer, which significantly enhances the ViT of \cite{dosovitskiy2020image} for encoding high…
VinVL: Revisiting Visual Representations in Vision-Language Models
Pengchuan Zhang, Xiujun Li, Xiaowei Hu +5
This paper presents a detailed study of improving visual representations for vision language (VL) tasks and develops an improved object detection model to provide object-centric re…
MiniVLM: A Smaller and Faster Vision-Language Model
Jianfeng Wang, Xiaowei Hu, Pengchuan Zhang +5
Recent vision-language (VL) studies have shown remarkable progress by learning generic representations from massive image-text pairs with transformer models and then fine-tuning on…
MagGAN: High-Resolution Face Attribute Editing with Mask-Guided Generative Adversarial Network
Yi Wei, Zhe Gan, Wenbo Li +5
We present Mask-guided Generative Adversarial Network (MagGAN) for high-resolution face attribute editing, in which semantic facial masks from a pre-trained face parser are used to…
Novel Human-Object Interaction Detection via Adversarial Domain Generalization
Yuhang Song, Wenbo Li, Lei Zhang +6
We study in this paper the problem of novel human-object interaction (HOI) detection, aiming at improving the generalization ability of the model to unseen scenarios. The challenge…
Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks
Xiujun Li, Xi Yin, Chunyuan Li +9
Large-scale pre-training methods of learning cross-modal representations on image-text pairs are becoming popular for vision-language tasks. While existing methods simply concatena…
TIGEr: Text-to-Image Grounding for Image Caption Evaluation
Ming Jiang, Qiuyuan Huang, Lei Zhang +5
This paper presents a new metric called TIGEr for the automatic evaluation of image captioning systems. Popular metrics, such as BLEU and CIDEr, are based solely on text matching b…
Object-driven Text-to-Image Synthesis via Adversarial Training
Wenbo Li, Pengchuan Zhang, Lei Zhang +4
In this paper, we propose Object-driven Attentive Generative Adversarial Newtorks (Obj-GANs) that allow object-centered text-to-image synthesis for complex scenes. Following the tw…
Towards Coherent and Cohesive Long-form Text Generation
Woon Sang Cho, Pengchuan Zhang, Yizhe Zhang +5
Generating coherent and cohesive long-form texts is a challenging task. Previous works relied on large amounts of human-generated texts to train neural language models. However, fe…