A Battle of Network Structures: An Empirical Study of CNN, Transformer, and MLP
arXiv:2108.13002
Abstract
Convolutional neural networks (CNN) are the dominant deep neural network (DNN) architecture for computer vision. Recently, Transformer and multi-layer perceptron (MLP)-based models, such as Vision Transformer and MLP-Mixer, started to lead new trends as they showed promising results in the ImageNet classification task. In this paper, we conduct empirical studies on these DNN structures and try to understand their respective pros and cons. To ensure a fair comparison, we first develop a unified framework called SPACH which adopts separate modules for spatial and channel processing. Our experiments under the SPACH framework reveal that all structures can achieve competitive performance at a moderate scale. However, they demonstrate distinctive behaviors when the network size scales up. Based on our findings, we propose two hybrid models using convolution and Transformer modules. The resulting Hybrid-MS-S+ model achieves 83.9% top-1 accuracy with 63M parameters and 12.3G FLOPS. It is already on par with the SOTA models with sophisticated designs. The code and models are publicly available at https://github.com/microsoft/SPACH.
References in corpus (11)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- MLP-Mixer: An all-MLP Architecture for Vision
- EfficientNetV2: Smaller Models and Faster Training
- Conditional Positional Encodings for Vision Transformers
- LocalViT: Analyzing Locality in Vision Transformers
- CvT: Introducing Convolutions to Vision Transformers
- All Tokens Matter: Token Labeling for Training Better Vision Transformers
- LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference
- Do You Even Need Attention? A Stack of Feed-Forward Layers Does Surprisingly Well on ImageNet
- MultiGrain: a unified image embedding for classes and instances
- Global Filter Networks for Image Classification