Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks
arXiv:2112.01522
Abstract
Biological intelligence systems of animals perceive the world by integrating information in different modalities and processing simultaneously for various tasks. In contrast, current machine learning research follows a task-specific paradigm, leading to inefficient collaboration between tasks and high marginal costs of developing perception models for new tasks. In this paper, we present a generic perception architecture named Uni-Perceiver, which processes a variety of modalities and tasks with unified modeling and shared parameters. Specifically, Uni-Perceiver encodes different task inputs and targets from arbitrary modalities into a unified representation space with a modality-agnostic Transformer encoder and lightweight modality-specific tokenizers. Different perception tasks are modeled as the same formulation, that is, finding the maximum likelihood target for each input through the similarity of their representations. The model is pre-trained on several uni-modal and multi-modal tasks, and evaluated on a variety of downstream tasks, including novel tasks that did not appear in the pre-training stage. Results show that our pre-trained model without any tuning can achieve reasonable performance even on novel tasks. The performance can be improved to a level close to state-of-the-art methods by conducting prompt tuning on 1% of downstream task data. Full-data fine-tuning further delivers results on par with or better than state-of-the-art results. Code shall be released.
References in corpus (25)
- Adam: A Method for Stochastic Optimization
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- mixup: Beyond Empirical Risk Minimization
- Learning to Prompt for Vision-Language Models
- The Kinetics Human Action Video Dataset
- ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
- Microsoft COCO Captions: Data Collection and Evaluation Server
- Pre-trained Models for Natural Language Processing: A Survey
- Is Space-Time Attention All You Need for Video Understanding?
- VisualBERT: A Simple and Performant Baseline for Vision and Language
- VL-BERT: Pre-training of Generic Visual-Linguistic Representations
- FractalNet: Ultra-Deep Neural Networks without Residuals
- ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision
- Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing
- A Short Note on the Kinetics-700-2020 Human Action Dataset
- DeepViT: Towards Deeper Vision Transformer
- VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text
- Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers
- P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks
- Learning Video Representations using Contrastive Bidirectional Transformer
- ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data
- CLIP2Video: Mastering Video-Text Retrieval via Image CLIP
- Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling
- CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval
- CLIP-Adapter: Better Vision-Language Models with Feature Adapters