EfficientNetV2: Smaller Models and Faster Training

Abstract

This paper introduces EfficientNetV2, a new family of convolutional networks that have faster training speed and better parameter efficiency than previous models. To develop this family of models, we use a combination of training-aware neural architecture search and scaling, to jointly optimize training speed and parameter efficiency. The models were searched from the search space enriched with new ops such as Fused-MBConv. Our experiments show that EfficientNetV2 models train much faster than state-of-the-art models while being up to 6.8x smaller. Our training can be further sped up by progressively increasing the image size during training, but it often causes a drop in accuracy. To compensate for this accuracy drop, we propose to adaptively adjust regularization (e.g., dropout and data augmentation) as well, such that we can achieve both fast training and good accuracy. With progressive learning, our EfficientNetV2 significantly outperforms previous models on ImageNet and CIFAR/Cars/Flowers datasets. By pretraining on the same ImageNet21k, our EfficientNetV2 achieves 87.3% top-1 accuracy on ImageNet ILSVRC2012, outperforming the recent ViT by 2.0% accuracy while training 5x-11x faster using the same computing resources. Code will be available at https://github.com/google/automl/tree/master/efficientnetv2.

arXiv commentICML 2021

References in corpus (11)

• Neural Architecture Search: A Survey
• MixConv: Mixed Depthwise Convolutional Kernels
• High-Performance Large-Scale Image Recognition Without Normalization
• Revisiting ResNets: Improved Training and Scaling Strategies
• Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation
• Fixing the train-test resolution discrepancy: FixEfficientNet
• Accelerator-aware Neural Network Design using AutoML
• LambdaNetworks: Modeling Long-Range Interactions Without Attention
• MobileDets: Searching for Object Detection Architectures for Mobile Accelerators
• Fixing the train-test resolution discrepancy
• PDA: Progressive Data Augmentation for General Robustness of Deep Neural Networks

Cited by in corpus (81)

• On the Opportunities and Risks of Foundation Models
• CoAtNet: Marrying Convolution and Attention for All Data Sizes
• A Survey on Ensemble Learning under the Era of Deep Learning
• Waste detection in Pomerania: non-profit project for detecting waste in environment
• Large AI Models in Health Informatics: Applications, Challenges, and the Future
• Compute and Energy Consumption Trends in Deep Learning Inference
• Tiny Machine Learning: Progress and Futures
• LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference
• AADG: Automatic Augmentation for Domain Generalization on Retinal Image Segmentation
• A Battle of Network Structures: An Empirical Study of CNN, Transformer, and MLP
• Real-time Neural Network Inference on Extremely Weak Devices: Agile Offloading with Explainable AI
• How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers
• SparseSwin: Swin Transformer with Sparse Transformer Block
• Cross-Forgery Analysis of Vision Transformers and CNNs for Deepfake Image Detection
• Vision Transformers with Patch Diversification
• Container: Context Aggregation Network
• TRILLsson: Distilled Universal Paralinguistic Speech Representations
• Danish Fungi 2020 -- Not Just Another Image Recognition Dataset
• Leveraging TCN and Transformer for effective visual-audio fusion in continuous emotion recognition
• Assaying Out-Of-Distribution Generalization in Transfer Learning
• Automated Human Cell Classification in Sparse Datasets using Few-Shot Learning
• EfficientTDNN: Efficient Architecture Search for Speaker Recognition
• A Cloud-based Deep Learning Framework for Early Detection of Pushing at Crowded Event Entrances
• Exploring Advances in Transformers and CNN for Skin Lesion Diagnosis on Small Datasets
• ROCT-Net: A new ensemble deep convolutional model with improved spatial resolution learning for detecting common diseases from retinal OCT images
• Leveraging Automatic Personalised Nutrition: Food Image Recognition Benchmark and Dataset based on Nutrition Taxonomy
• Homogeneous vector bundles and $G$-equivariant convolutional neural networks
• Hand bone age estimation using divide and conquer strategy and lightweight convolutional neural networks
• VEDLIoT: Very Efficient Deep Learning in IoT
• ThreshNet: An Efficient DenseNet Using Threshold Mechanism to Reduce Connections
• Hard-Attention for Scalable Image Classification
• Edge Devices Inference Performance Comparison
• Rethinking Dilated Convolution for Real-time Semantic Segmentation
• LeYOLO, New Embedded Architecture for Object Detection
• Deep Learning for Pavement Condition Evaluation Using Satellite Imagery
• AnalogNets: ML-HW Co-Design of Noise-robust TinyML Models and Always-On Analog Compute-in-Memory Accelerator
• Making EfficientNet More Efficient: Exploring Batch-Independent Normalization, Group Convolutions and Reduced Resolution Training
• Deep Learning and LLM-based Methods Applied to Stellar Lightcurve Classification
• Neural Architecture Search using Property Guided Synthesis
• Robustness of Visual Explanations to Common Data Augmentation
• SinkSAM-Net: Knowledge-Driven Self-Supervised Sinkhole Segmentation Using Topographic Priors and Segment Anything Model
• Ensemble Neural Networks for Remaining Useful Life (RUL) Prediction
• Forged Image Detection using SOTA Image Classification Deep Learning Methods for Image Forensics with Error Level Analysis
• Task-Adaptive Neural Network Search with Meta-Contrastive Learning
• A Medical Pre-Diagnosis System for Histopathological Image of Breast Cancer
• Detection-segmentation convolutional neural network for autonomous vehicle perception
• Detection of retinal diseases using an accelerated reused convolutional network
• DKM: Differentiable K-Means Clustering Layer for Neural Network Compression
• Slimmable Encoders for Flexible Split DNNs in Bandwidth and Resource Constrained IoT Systems
• A Winning Hand: Compressing Deep Networks Can Improve Out-Of-Distribution Robustness
• Picking Up Quantization Steps for Compressed Image Classification
• Learning from the few: Fine-grained approach to pediatric wrist pathology recognition on a limited dataset
• Fast TILs -- A Pipeline for Efficient TILs Estimation in Non-Small Cell Lung Cancer
• 3rd Place Solution to Google Landmark Recognition Competition 2021
• Investigating Transfer Learning Capabilities of Vision Transformers and CNNs by Fine-Tuning a Single Trainable Block
• Recognizing bird species in diverse soundscapes under weak supervision
• EnTri: Ensemble Learning with Tri-level Representations for Explainable Scene Recognition
• UniNet: Unified Architecture Search with Convolution, Transformer, and MLP
• Concurrent Neural Tree and Data Preprocessing AutoML for Image Classification
• Deep Learning Applications Based on WISE Infrared Data: Classification of Stars, Galaxies and Quasars
• Benchmarking Deep Learning Methods for Irradiance Estimation from Sky Images with Applications to Video Prediction-Based Irradiance Nowcasting
• Navigating limitations with precision: A fine-grained ensemble approach to wrist pathology recognition on a limited x-ray dataset
• Zero-Cost Operation Scoring in Differentiable Architecture Search
• Two-Stream Thermal Imaging Fusion for Enhanced Time of Birth Detection in Neonatal Care
• Femtosecond pulse parameter estimation from photoelectron momenta using machine learning
• vieCap4H-VLSP 2021: Vietnamese Image Captioning for Healthcare Domain using Swin Transformer and Attention-based LSTM
• RaftMLP: How Much Can Be Done Without Attention and with Less Spatial Locality?
• Greedy Network Enlarging
• Dense Pruning of Pointwise Convolutions in the Frequency Domain
• Google Landmark Retrieval 2021 Competition Third Place Solution
• Phantom: A High-Performance Computational Core for Sparse Convolutional Neural Networks
• Solution for Large-scale Long-tailed Recognition with Noisy Labels
• NoFADE: Analyzing Diminishing Returns on CO2 Investment
• Exploring and Improving Mobile Level Vision Transformers
• LAE : Long-tailed Age Estimation
• DEM Super-Resolution with EfficientNetV2
• Improved Robustness of Vision Transformer via PreLayerNorm in Patch Embedding
• At First Contact: Stiffness Estimation Using Vibrational Information for Prosthetic Grasp Modulation
• 3rd Place Solution for NeurIPS 2021 Shifts Challenge: Vehicle Motion Prediction
• Neural network scoring for efficient computing
• Vision Pair Learning: An Efficient Training Framework for Image Classification