Towards Better Accuracy-efficiency Trade-offs: Divide and Co-training

Abstract

The width of a neural network matters since increasing the width will necessarily increase the model capacity. However, the performance of a network does not improve linearly with the width and soon gets saturated. In this case, we argue that increasing the number of networks (ensemble) can achieve better accuracy-efficiency trade-offs than purely increasing the width. To prove it, one large network is divided into several small ones regarding its parameters and regularization components. Each of these small networks has a fraction of the original one's parameters. We then train these small networks together and make them see various views of the same data to increase their diversity. During this co-training process, networks can also learn from each other. As a result, small networks can achieve better ensemble performance than the large one with few or no extra parameters or FLOPs, \ie, achieving better accuracy-efficiency trade-offs. Small networks can also achieve faster inference speed than the large one by concurrent running. All of the above shows that the number of networks is a new dimension of model scaling. We validate our argument with 8 different neural architectures on common benchmarks through extensive experiments. The code is available at \url{https://github.com/FreeformRobotics/Divide-and-Co-training}.

arXiv commentaccepted by T-IP 2022, code is at https://github.com/FreeformRobotics/Divide-and-Co-training

References in corpus (32)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• SSD: Single Shot MultiBox Detector
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Distilling the Knowledge in a Neural Network
• Rethinking Atrous Convolution for Semantic Image Segmentation
• Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation
• mixup: Beyond Empirical Risk Minimization
• Neural Architecture Search with Reinforcement Learning
• Improved Regularization of Convolutional Neural Networks with Cutout
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• MobileNetV2: Inverted Residuals and Linear Bottlenecks
• Squeeze-and-Excitation Networks
• Temporal Ensembling for Semi-Supervised Learning
• DARTS: Differentiable Architecture Search
• Mixed Precision Training
• A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay
• Weighted boxes fusion: Ensembling boxes from different object detection models
• ResNeSt: Split-Attention Networks
• Mutual Mean-Teaching: Pseudo Label Refinery for Unsupervised Domain Adaptation on Person Re-identification
• Shake-Shake regularization
• Slimmable Neural Networks
• ShakeDrop Regularization for Deep Residual Learning
• Bag of Tricks for Image Classification with Convolutional Neural Networks
• Snapshot Ensembles: Train 1, get M for free
• Collaborative Learning for Deep Neural Networks
• Model Rubik's Cube: Twisting Resolution, Depth and Width for TinyNets
• MultiGrain: a unified image embedding for classes and instances
• When Ensembling Smaller Models is More Efficient than Single Large Models
• Time Matters in Regularizing Deep Networks: Weight Decay and Data Augmentation Affect Early Learning Dynamics, Matter Little Near Convergence
• Cooperative Learning with Visual Attributes
• Deep Neural Network Ensembles

Cited by in corpus (3)

• FedDCT: Federated Learning of Large Convolutional Neural Networks on Resource Constrained Devices using Divide and Collaborative Training
• Robust Learning Meets Generative Models: Can Proxy Distributions Improve Adversarial Robustness?
• Generalized Data Weighting via Class-level Gradient Manipulation