699 citations · 3.7k across the 65 of their papers we have counts for
69 papers
Large Multimodal Models: Notes on CVPR 2023 Tutorial
Chunyuan Li
This tutorial note summarizes the presentation on ``Large Multimodal Models: Towards Building and Surpassing Multimodal GPT-4'', a part of CVPR 2023 tutorial on ``Recent Advances i…
LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day
Chunyuan Li, Cliff Wong, Sheng Zhang +6
Conversational generative AI has demonstrated remarkable promise for empowering biomedical practitioners, but current investigations focus on unimodal text. Multimodal conversation…
Towards Building the Federated GPT: Federated Instruction Tuning
Jianyi Zhang, Saeed Vahidian, Martin Kuo +6
While "instruction-tuned" generative large language models (LLMs) have demonstrated an impressive ability to generalize to new tasks, the training phases heavily rely on large amou…
Instruction Tuning with GPT-4
Baolin Peng, Chunyuan Li, Pengcheng He +2
Prior work has shown that finetuning large language models (LLMs) using machine-generated instruction-following data enables such models to achieve remarkable zero-shot capabilitie…
Visual Instruction Tuning
Haotian Liu, Chunyuan Li, Qingyang Wu +1
Instruction tuning large language models (LLMs) using machine-generated instruction-following data has improved zero-shot capabilities on new tasks, but the idea is less explored i…
Scaling Vision-Language Models with Sparse Mixture of Experts
Sheng Shen, Zhewei Yao, Chunyuan Li +3
The field of natural language processing (NLP) has made significant strides in recent years, particularly in the development of large-scale vision-language models (VLMs). These mod…
Learning Customized Visual Models with Retrieval-Augmented Knowledge
Haotian Liu, Kilho Son, Jianwei Yang +4
Image-text contrastive learning models such as CLIP have demonstrated strong task transfer ability. The high generality and usability of these visual models is achieved via a web-s…
GLIGEN: Open-Set Grounded Text-to-Image Generation
Yuheng Li, Haotian Liu, Qingyang Wu +5
Large-scale text-to-image diffusion models have made amazing advances. However, the status quo is to use text input alone, which can impede controllability. In this work, we propos…
Generalized Decoding for Pixel, Image, and Language
Xueyan Zou, Zi-Yi Dou, Jianwei Yang +11
We present X-Decoder, a generalized decoding model that can predict pixel-level segmentation and language tokens seamlessly. X-Decodert takes as input two types of queries: (i) gen…
STT: Soft Template Tuning for Few-Shot Adaptation
Ping Yu, Wei Wang, Chunyuan Li +3
Prompt tuning has been an extremely effective tool to adapt a pre-trained model to downstream tasks. However, standard prompt-based methods mainly consider the case of sufficient d…
K-LITE: Learning Transferable Visual Models with External Knowledge
Sheng Shen, Chunyuan Li, Xiaowei Hu +11
The new generation of state-of-the-art computer vision systems are trained from natural language supervision, ranging from simple object category names to descriptive captions. Thi…
Parameter-efficient Model Adaptation for Vision Transformers
Xuehai He, Chunyuan Li, Pengchuan Zhang +2
In computer vision, it has achieved great transfer learning performance via adapting large-scale pretrained vision models (e.g., vision transformers) to downstream tasks. Common ap…
A Generic Approach for Enhancing GANs by Regularized Latent Optimization
Yufan Zhou, Chunyuan Li, Changyou Chen +1
With the rapidly growing model complexity and data volume, training deep generative models (DGMs) for better performance has becoming an increasingly more important challenge. Prev…
Grounded Language-Image Pre-training
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang +9
This paper presents a grounded language-image pre-training (GLIP) model for learning object-level, language-aware, and semantic-rich visual representations. GLIP unifies object det…
Florence: A New Foundation Model for Computer Vision
Lu Yuan, Dongdong Chen, Yi-Ling Chen +20
Automated visual understanding of our diverse and open world demands computer vision models to generalize well with minimal customization for specific tasks, similar to human visio…
LAFITE: Towards Language-Free Training for Text-to-Image Generation
Yufan Zhou, Ruiyi Zhang, Changyou Chen +6
One of the major challenges in training text-to-image generation models is the need of a large number of high-quality image-text pairs. While image samples are often easily accessi…
SYNERGY: Building Task Bots at Scale Using Symbolic Knowledge and Machine Teaching
Baolin Peng, Chunyuan Li, Zhu Zhang +3
In this paper we explore the use of symbolic knowledge and machine teaching to reduce human data labeling efforts in building neural task bots. We propose SYNERGY, a hybrid learnin…
Focal Self-attention for Local-Global Interactions in Vision Transformers
Jianwei Yang, Chunyuan Li, Pengchuan Zhang +4
Recently, Vision Transformer and its variants have shown great promise on various computer vision tasks. The ability of capturing short- and long-range visual dependencies through…
Efficient Self-supervised Vision Transformers for Representation Learning
Chunyuan Li, Jianwei Yang, Pengchuan Zhang +5
This paper investigates two techniques for developing efficient self-supervised vision transformers (EsViT) for visual representation learning. First, we show through a comprehensi…
Exploring Robustness of Unsupervised Domain Adaptation in Semantic Segmentation
Jinyu Yang, Chunyuan Li, Weizhi An +5
Recent studies imply that deep neural networks are vulnerable to adversarial examples -- inputs with a slight but intentional perturbation are incorrectly classified by the network…
Contrastive Attraction and Contrastive Repulsion for Representation Learning
Huangjie Zheng, Xu Chen, Jiangchao Yao +7
Contrastive learning (CL) methods effectively learn data representations in a self-supervision manner, where the encoder contrasts each positive sample over multiple negative sampl…
Partition-Guided GANs
Mohammadreza Armandpour, Ali Sadeghian, Chunyuan Li +1
Despite the success of Generative Adversarial Networks (GANs), their training suffers from several well-known problems, including mode collapse and difficulties learning a disconne…
Self-supervised Pre-training with Hard Examples Improves Visual Representations
Chunyuan Li, Xiujun Li, Lei Zhang +3
Self-supervised pre-training (SSP) employs random image transformations to generate training data for visual representation learning. In this paper, we first present a modeling fra…
SDA: Improving Text Generation with Self Data Augmentation
Ping Yu, Ruiyi Zhang, Yang Zhao +3
Data augmentation has been widely used to improve deep neural networks in many research fields, such as computer vision. However, less work has been done in the context of text, pa…