Florence: A New Foundation Model for Computer Vision

Abstract

Automated visual understanding of our diverse and open world demands computer vision models to generalize well with minimal customization for specific tasks, similar to human vision. Computer vision foundation models, which are trained on diverse, large-scale dataset and can be adapted to a wide range of downstream tasks, are critical for this mission to solve real-world computer vision applications. While existing vision foundation models such as CLIP, ALIGN, and Wu Dao 2.0 focus mainly on mapping images and textual representations to a cross-modal shared representation, we introduce a new computer vision foundation model, Florence, to expand the representations from coarse (scene) to fine (object), from static (images) to dynamic (videos), and from RGB to multiple modalities (caption, depth). By incorporating universal visual-language representations from Web-scale image-text data, our Florence model can be easily adapted for various computer vision tasks, such as classification, retrieval, object detection, VQA, image caption, video retrieval and action recognition. Moreover, Florence demonstrates outstanding performance in many types of transfer learning: fully sampled fine-tuning, linear probing, few-shot transfer and zero-shot transfer for novel images and objects. All of these properties are critical for our vision foundation model to serve general purpose vision tasks. Florence achieves new state-of-the-art results in majority of 44 representative benchmarks, e.g., ImageNet-1K zero-shot classification with top-1 accuracy of 83.74 and the top-5 accuracy of 97.18, 62.4 mAP on COCO fine tuning, 80.36 on VQA, and 87.8 on Kinetics-600.

References in corpus (27)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• A Simple Framework for Contrastive Learning of Visual Representations
• Language Models are Few-Shot Learners
• On the Opportunities and Risks of Foundation Models
• Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
• Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)
• Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
• CoAtNet: Marrying Convolution and Attention for All Data Sizes
• Big Self-Supervised Models are Strong Semi-Supervised Learners
• SimVLM: Simple Visual Language Model Pretraining with Weak Supervision
• Large-Scale Adversarial Training for Vision-and-Language Representation Learning
• Focal Self-attention for Local-Global Interactions in Vision Transformers
• Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
• Self-Supervised MultiModal Versatile Networks
• CvT: Introducing Convolutions to Vision Transformers
• ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data
• ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph
• Dynamic Head: Unifying Object Detection Heads with Attentions
• VinVL: Revisiting Visual Representations in Vision-Language Models
• Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models
• Video Swin Transformer
• MiniVLM: A Smaller and Faster Vision-Language Model
• Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning
• Learning the Best Pooling Strategy for Visual Semantic Embedding
• An Empirical Study of Training End-to-End Vision-and-Language Transformers
• TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?
• Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup

Cited by in corpus (14)

• Learning to Prompt for Vision-Language Models
• Large AI Models in Health Informatics: Applications, Challenges, and the Future
• CenterCLIP: Token Clustering for Efficient Text-Video Retrieval
• Self-supervised remote sensing feature learning: Learning Paradigms, Challenges, and Future Works
• From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunities
• CLIP4STR: A Simple Baseline for Scene Text Recognition with Pre-trained Vision-Language Model
• FoodSAM: Any Food Segmentation
• Multimodal Foundation Models for Material Property Prediction and Discovery
• From Screens to Scenes: A Survey of Embodied AI in Healthcare
• FeatureNeRF: Learning Generalizable NeRFs by Distilling Foundation Models
• An Empirical Study of Training End-to-End Vision-and-Language Transformers
• Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning
• Compositional Kronecker Context Optimization for Vision-Language Models
• A Good Foundation is Worth Many Labels: Label-Efficient Panoptic Segmentation