Collaboration of Experts: Achieving 80% Top-1 Accuracy on ImageNet with 100M FLOPs
arXiv:2107.03815
Abstract
In this paper, we propose a Collaboration of Experts (CoE) framework to pool together the expertise of multiple networks towards a common aim. Each expert is an individual network with expertise on a unique portion of the dataset, which enhances the collective capacity. Given a sample, an expert is selected by the delegator, which simultaneously outputs a rough prediction to support early termination. To fulfill this framework, we propose three modules to impel each model to play its role, namely weight generation module (WGM), label generation module (LGM) and variance calculation module (VCM). Our method achieves the state-of-the-art performance on ImageNet, 80.7% top-1 accuracy with 194M FLOPs. Combined with PWLU activation function and CondConv, CoE further achieves the accuracy of 80.0% with only 100M FLOPs for the first time. More importantly, our method is hardware friendly and achieves a 3-6x speedup compared with some existing conditional computation approaches.
ICML
References in corpus (13)
- Distilling the Knowledge in a Neural Network
- Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
- Addressing Failure Prediction by Learning Model Confidence
- Adversarial AutoAugment
- Are we done with ImageNet?
- DyNet: Dynamic Convolution for Accelerating Convolutional Neural Networks
- Model Rubik's Cube: Twisting Resolution, Depth and Width for TinyNets
- Training independent subnetworks for robust prediction
- Glance and Focus: a Dynamic Approach to Reducing Spatial Redundancy in Image Classification
- BETANAS: BalancEd TrAining and selective drop for Neural Architecture Search
- WeightNet: Revisiting the Design Space of Weight Networks
- Dynamic Graph: Learning Instance-aware Connectivity for Neural Networks
- BasisNet: Two-stage Model Synthesis for Efficient Inference