Depth-Wise Convolutions in Vision Transformers for Efficient Training on Small Datasets

Abstract

The Vision Transformer (ViT) leverages the Transformer's encoder to capture global information by dividing images into patches and achieves superior performance across various computer vision tasks. However, the self-attention mechanism of ViT captures the global context from the outset, overlooking the inherent relationships between neighboring pixels in images or videos. Transformers mainly focus on global information while ignoring the fine-grained local details. Consequently, ViT lacks inductive bias during image or video dataset training. In contrast, convolutional neural networks (CNNs), with their reliance on local filters, possess an inherent inductive bias, making them more efficient and quicker to converge than ViT with less data. In this paper, we present a lightweight Depth-Wise Convolution module as a shortcut in ViT models, bypassing entire Transformer blocks to ensure the models capture both local and global information with minimal overhead. Additionally, we introduce two architecture variants, allowing the Depth-Wise Convolution modules to be applied to multiple Transformer blocks for parameter savings, and incorporating independent parallel Depth-Wise Convolution modules with different kernels to enhance the acquisition of local information. The proposed approach significantly boosts the performance of ViT models on image classification, object detection, and instance segmentation by a large margin, especially on small datasets, as evaluated on CIFAR-10, CIFAR-100, Tiny-ImageNet and ImageNet for image classification, and COCO for object detection and instance segmentation. The source code can be accessed at https://github.com/ZTX-100/Efficient_ViT_with_DW.

References in corpus (15)

• Adam: A Method for Stochastic Optimization
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• mixup: Beyond Empirical Risk Minimization
• Gaussian Error Linear Units (GELUs)
• Twins: Revisiting the Design of Spatial Attention in Vision Transformers
• Conditional Positional Encodings for Vision Transformers
• LocalViT: Analyzing Locality in Vision Transformers
• ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias
• Shuffle Transformer: Rethinking Spatial Shuffle for Vision Transformer
• Talking-Heads Attention
• Bridging the Gap Between Vision Transformers and Convolutional Neural Networks on Small Datasets
• PSViT: Better Vision Transformer via Token Pooling and Attention Sharing
• Miti-DETR: Object Detection based on Transformers with Mitigatory Self-Attention Convergence
• DiT: Efficient Vision Transformers with Dynamic Token Routing
• SuperLoRA: Parameter-Efficient Unified Adaptation of Multi-Layer Attention Modules