Escaping the Big Data Paradigm with Compact Transformers

Abstract

With the rise of Transformers as the standard for language processing, and their advancements in computer vision, there has been a corresponding growth in parameter size and amounts of training data. Many have come to believe that because of this, transformers are not suitable for small sets of data. This trend leads to concerns such as: limited availability of data in certain scientific domains and the exclusion of those with limited resource from research in the field. In this paper, we aim to present an approach for small-scale learning by introducing Compact Transformers. We show for the first time that with the right size, convolutional tokenization, transformers can avoid overfitting and outperform state-of-the-art CNNs on small datasets. Our models are flexible in terms of model size, and can have as little as 0.28M parameters while achieving competitive results. Our best model can reach 98% accuracy when training from scratch on CIFAR-10 with only 3.7M parameters, which is a significant improvement in data-efficiency over previous Transformer based models being over 10x smaller than other transformers and is 15% the size of ResNet50 while achieving similar performance. CCT also outperforms many modern CNN based approaches, and even some recent NAS-based approaches. Additionally, we obtain a new SOTA result on Flowers-102 with 99.76% top-1 accuracy, and improve upon the existing baseline on ImageNet (82.71% accuracy with 29% as many parameters as ViT), as well as NLP tasks. Our simple and compact design for transformers makes them more feasible to study for those with limited computing resources and/or dealing with small datasets, while extending existing research efforts in data efficient transformers. Our code and pre-trained models are publicly available at https://github.com/SHI-Labs/Compact-Transformers.

arXiv commentAdded new results on Flowers-102, distillation

References in corpus (8)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Deep Learning in Neural Networks: An Overview
• mixup: Beyond Empirical Risk Minimization
• Self-Attention Generative Adversarial Networks
• Is Space-Time Attention All You Need for Video Understanding?
• Effective Approaches to Attention-based Neural Machine Translation
• Stand-Alone Self-Attention in Vision Models
• CvT: Introducing Convolutions to Vision Transformers

Cited by in corpus (30)

• Transformers in Vision: A Survey
• TriTransNet: RGB-D Salient Object Detection with a Triplet Transformer Embedding Network
• Shuffle Transformer: Rethinking Spatial Shuffle for Vision Transformer
• Recent Progress in Transformer-based Medical Image Analysis
• An Experimental Study of Byzantine-Robust Aggregation Schemes in Federated Learning
• Improvement of Performance in Freezing of Gait detection in Parkinsons Disease using Transformer networks and a single waist worn triaxial accelerometer
• Clinically-Inspired Multi-Agent Transformers for Disease Trajectory Forecasting from Multimodal Data
• Enhancing MRI-Based Classification of Alzheimer's Disease with Explainable 3D Hybrid Compact Convolutional Transformers
• ConvMLP: Hierarchical Convolutional MLPs for Vision
• TransFusionOdom: Interpretable Transformer-based LiDAR-Inertial Fusion Odometry Estimation
• ITA: An Energy-Efficient Attention and Softmax Accelerator for Quantized Transformers
• Stochastic-Sign SGD for Federated Learning with Theoretical Guarantees
• Synthesized Speech Detection Using Convolutional Transformer-Based Spectrogram Analysis
• Fourier-basis Functions to Bridge Augmentation Gap: Rethinking Frequency Augmentation in Image Classification
• OnDev-LCT: On-Device Lightweight Convolutional Transformers towards federated learning
• Union-net: A deep neural network model adapted to small data sets
• Learning Sequence Descriptor based on Spatio-Temporal Attention for Visual Place Recognition
• Vision Xformers: Efficient Attention for Image Classification
• UFO-ViT: High Performance Linear Vision Transformer without Softmax
• Activation Function Optimization Scheme for Image Classification
• Triggering Dark Showers with Conditional Dual Auto-Encoders
• Full-attention based Neural Architecture Search using Context Auto-regression
• MSN: Efficient Online Mask Selection Network for Video Instance Segmentation
• Non-Intrusive Binaural Speech Intelligibility Prediction from Discrete Latent Representations
• Harnessing Orthogonality to Train Low-Rank Neural Networks
• Hybrid BYOL-ViT: Efficient approach to deal with small datasets
• Audiomer: A Convolutional Transformer For Keyword Spotting
• Noisy Feature Mixup
• Contrastive Masked Autoencoders for Character-Level Open-Set Writer Identification
• CpT: Convolutional Point Transformer for 3D Point Cloud Processing