ConTNet: Why not use convolution and transformer at the same time?

Abstract

Although convolutional networks (ConvNets) have enjoyed great success in computer vision (CV), it suffers from capturing global information crucial to dense prediction tasks such as object detection and segmentation. In this work, we innovatively propose ConTNet (ConvolutionTransformer Network), combining transformer with ConvNet architectures to provide large receptive fields. Unlike the recently-proposed transformer-based models (e.g., ViT, DeiT) that are sensitive to hyper-parameters and extremely dependent on a pile of data augmentations when trained from scratch on a midsize dataset (e.g., ImageNet1k), ConTNet can be optimized like normal ConvNets (e.g., ResNet) and preserve an outstanding robustness. It is also worth pointing that, given identical strong data augmentations, the performance improvement of ConTNet is more remarkable than that of ResNet. We present its superiority and effectiveness on image classification and downstream tasks. For example, our ConTNet achieves 81.8% top-1 accuracy on ImageNet which is the same as DeiT-B with less than 40% computational complexity. ConTNet-M also outperforms ResNet50 as the backbone of both Faster-RCNN (by 2.6%) and Mask-RCNN (by 3.2%) on COCO2017 dataset. We hope that ConTNet could serve as a useful backbone for CV tasks and bring new ideas for model design

References in corpus (3)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Stand-Alone Self-Attention in Vision Models

Cited by in corpus (9)

• RegionViT: Regional-to-Local Attention for Vision Transformers
• IA-RED${}^2$: Interpretability-Aware Redundancy Reduction for Vision Transformers
• CMT: Convolutional Neural Networks Meet Vision Transformers
• Dual-stream Network for Visual Recognition
• PSLT: A Light-weight Vision Transformer with Ladder Self-Attention and Progressive Shift
• Foundation Models and Transformers for Anomaly Detection: A Survey
• On the Integration of Self-Attention and Convolution
• Ripple Attention for Visual Perception with Sub-quadratic Complexity
• Exploring and Improving Mobile Level Vision Transformers