E${}^2$CM: Early Exit via Class Means for Efficient Supervised and Unsupervised Learning

Abstract

State-of-the-art neural networks with early exit mechanisms often need considerable amount of training and fine tuning to achieve good performance with low computational cost. We propose a novel early exit technique, Early Exit Class Means (E${}^2$CM), based on class means of samples. Unlike most existing schemes, E${}^2$CM does not require gradient-based training of internal classifiers and it does not modify the base network by any means. This makes it particularly useful for neural network training in low-power devices, as in wireless edge networks. We evaluate the performance and overheads of E${}^2$CM over various base neural networks such as MobileNetV3, EfficientNet, ResNet, and datasets such as CIFAR-100, ImageNet, and KMNIST. Our results show that, given a fixed training time budget, E${}^2$CM achieves higher accuracy as compared to existing early exit mechanisms. Moreover, if there are no limitations on the training time budget, E${}^2$CM can be combined with an existing early exit scheme to boost the latter's performance, achieving a better trade-off between computational cost and network accuracy. We also show that E${}^2$CM can be used to decrease the computational cost in unsupervised learning tasks.

arXiv comment8 pages, 4 figures, 2 tables. Accepted to IJCNN 2022 (WCCI2022)

References in corpus (9)

• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Prototypical Networks for Few-shot Learning
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Prevalence of Neural Collapse during the terminal phase of deep learning training
• Adaptive Neural Networks for Efficient Inference
• Shallow-Deep Networks: Understanding and Mitigating Network Overthinking
• Deciding How to Decide: Dynamic Routing in Artificial Neural Networks
• Separation and Concentration in Deep Networks