CMT: Convolutional Neural Networks Meet Vision Transformers

Abstract

Vision transformers have been successfully applied to image recognition tasks due to their ability to capture long-range dependencies within an image. However, there are still gaps in both performance and computational cost between transformers and existing convolutional neural networks (CNNs). In this paper, we aim to address this issue and develop a network that can outperform not only the canonical transformers, but also the high-performance convolutional models. We propose a new transformer based hybrid network by taking advantage of transformers to capture long-range dependencies, and of CNNs to model local features. Furthermore, we scale it to obtain a family of models, called CMTs, obtaining much better accuracy and efficiency than previous convolution and transformer based models. In particular, our CMT-S achieves 83.5% top-1 accuracy on ImageNet, while being 14x and 2x smaller on FLOPs than the existing DeiT and EfficientNet, respectively. The proposed CMT-S also generalizes well on CIFAR10 (99.2%), CIFAR100 (91.7%), Flowers (98.7%), and other challenging vision datasets such as COCO (44.3% mAP), with considerably less computational cost.

arXiv commentAccepted in CVPR 2022

References in corpus (16)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Decoupled Weight Decay Regularization
• Gaussian Error Linear Units (GELUs)
• Deformable DETR: Deformable Transformers for End-to-End Object Detection
• Transformer in Transformer
• Gather-Excite: Exploiting Feature Context in Convolutional Neural Networks
• Conditional Positional Encodings for Vision Transformers
• Stand-Alone Self-Attention in Vision Models
• CvT: Introducing Convolutions to Vision Transformers
• Training Vision Transformers for Image Retrieval
• Pre-Trained Image Processing Transformer
• ConTNet: Why not use convolution and transformer at the same time?
• RelationNet++: Bridging Visual Representations for Object Detection via Transformer Decoder

Cited by in corpus (7)

• A Survey on Visual Transformer
• Chasing Sparsity in Vision Transformers: An End-to-End Exploration
• MetaFormer Is Actually What You Need for Vision
• OH-Former: Omni-Relational High-Order Transformer for Person Re-Identification
• Pruning Self-attentions into Convolutional Layers in Single Path
• SparX: A Sparse Cross-Layer Connection Mechanism for Hierarchical Vision Mamba and Transformer Networks
• Shunted Self-Attention via Multi-Scale Token Aggregation