Multimodal Few-Shot Learning with Frozen Language Models

Abstract

When trained at sufficient scale, auto-regressive language models exhibit the notable ability to learn a new language task after being prompted with just a few examples. Here, we present a simple, yet effective, approach for transferring this few-shot learning ability to a multimodal setting (vision and language). Using aligned image and caption data, we train a vision encoder to represent each image as a sequence of continuous embeddings, such that a pre-trained, frozen language model prompted with this prefix generates the appropriate caption. The resulting system is a multimodal few-shot learner, with the surprising ability to learn a variety of new tasks when conditioned on examples, represented as a sequence of multiple interleaved image and text embeddings. We demonstrate that it can rapidly learn words for new objects and novel visual categories, do visual question-answering with only a handful of examples, and make use of outside knowledge, by measuring a single model on a variety of established and new benchmarks.

References in corpus (14)

• Learning Transferable Visual Models From Natural Language Supervision
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• Language Models are Few-Shot Learners
• Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• VL-BERT: Pre-training of Generic Visual-Linguistic Representations
• REALM: Retrieval-Augmented Language Model Pre-Training
• Prefix-Tuning: Optimizing Continuous Prompts for Generation
• Towards a Human-like Open-Domain Chatbot
• Unifying Vision-and-Language Tasks via Text Generation
• Pretrained Transformers as Universal Computation Engines
• Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML
• VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning
• Latent Translation: Crossing Modalities by Bridging Generative Models

Cited by in corpus (14)

• On the Opportunities and Risks of Foundation Models
• VLP: A Survey on Vision-Language Pre-training
• Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey
• CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models
• Few-Shot Bot: Prompt-Based Learning for Dialogue Systems
• Framing the News:From Human Perception to Large Language Model Inferences
• Automatically Generating UI Code from Screenshot: A Divide-and-Conquer-Based Approach
• PREDILECT: Preferences Delineated with Zero-Shot Language-based Reasoning in Reinforcement Learning
• EfficientCLIP: Efficient Cross-Modal Pre-training by Ensemble Confident Learning and Language Modeling
• MMInstruct: A High-Quality Multi-Modal Instruction Tuning Dataset with Extensive Diversity
• KNN Transformer with Pyramid Prompts for Few-Shot Learning
• BEAMetrics: A Benchmark for Language Generation Evaluation Evaluation
• TEASEL: A Transformer-Based Speech-Prefixed Language Model
• Teaching Autoregressive Language Models Complex Tasks By Demonstration