MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer

Abstract

Light-weight convolutional neural networks (CNNs) are the de-facto for mobile vision tasks. Their spatial inductive biases allow them to learn representations with fewer parameters across different vision tasks. However, these networks are spatially local. To learn global representations, self-attention-based vision trans-formers (ViTs) have been adopted. Unlike CNNs, ViTs are heavy-weight. In this paper, we ask the following question: is it possible to combine the strengths of CNNs and ViTs to build a light-weight and low latency network for mobile vision tasks? Towards this end, we introduce MobileViT, a light-weight and general-purpose vision transformer for mobile devices. MobileViT presents a different perspective for the global processing of information with transformers, i.e., transformers as convolutions. Our results show that MobileViT significantly outperforms CNN- and ViT-based networks across different tasks and datasets. On the ImageNet-1k dataset, MobileViT achieves top-1 accuracy of 78.4% with about 6 million parameters, which is 3.2% and 6.2% more accurate than MobileNetv3 (CNN-based) and DeIT (ViT-based) for a similar number of parameters. On the MS-COCO object detection task, MobileViT is 5.7% more accurate than MobileNetv3 for a similar number of parameters. Our source code is open-source and available at: https://github.com/apple/ml-cvnets

Accepted at ICLR'22

References in corpus (12)

• Rethinking Atrous Convolution for Semantic Image Segmentation
• Linformer: Self-Attention with Linear Complexity
• CoAtNet: Marrying Convolution and Attention for All Data Sizes
• Early Convolutions Help Transformers See Better
• GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
• DeepViT: Towards Deeper Vision Transformer
• DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification
• LocalViT: Analyzing Locality in Vision Transformers
• CvT: Introducing Convolutions to Vision Transformers
• LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference
• Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions
• CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification

Cited by in corpus (5)

• Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better
• A Convolutional Vision Transformer for Semantic Segmentation of Side-Scan Sonar Data
• LeYOLO, New Embedded Architecture for Object Detection
• F3-Pruning: A Training-Free and Generalized Pruning Strategy towards Faster and Finer Text-to-Video Synthesis
• MultiTASC++: A Continuously Adaptive Scheduler for Edge-Based Multi-Device Cascade Inference