SIPA: A Simple Framework for Efficient Networks

Abstract

With the success of deep learning in various fields and the advent of numerous Internet of Things (IoT) devices, it is essential to lighten models suitable for low-power devices. In keeping with this trend, MicroNet Challenge, which is the challenge to build efficient models from the view of both storage and computation, was hosted at NeurIPS 2019. To develop efficient models through this challenge, we propose a framework, coined as SIPA, consisting of four stages: Searching, Improving, Pruning, and Accelerating. With the proposed framework, our team, OSI AI, compressed 334x the parameter storage and 357x the math operation compared to WideResNet-28-10 and took 4th place in the CIFAR-100 track at MicroNet Challenge 2019 with the top 10% highly efficient computation. Our source code is available from https://github.com/Lee-Gihun/MicroNet_OSI-AI.

arXiv comment14 pages, 9 figures

References in corpus (17)

• Distilling the Knowledge in a Neural Network
• mixup: Beyond Empirical Risk Minimization
• Neural Architecture Search with Reinforcement Learning
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• Squeeze-and-Excitation Networks
• Wide Residual Networks
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Pruning Convolutional Neural Networks for Resource Efficient Inference
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• Rethinking the Value of Network Pruning
• When Does Label Smoothing Help?
• Searching for Activation Functions
• To prune, or not to prune: exploring the efficacy of pruning for model compression
• Train longer, generalize better: closing the generalization gap in large batch training of neural networks
• Training Deep Neural Networks with 8-bit Floating Point Numbers
• Selective Classification for Deep Neural Networks
• Convolution with even-sized kernels and symmetric padding