Compacting Deep Neural Networks for Internet of Things: Methods and Applications
arXiv:2103.11083 · doi:10.1109/JIOT.2021.3063497
Abstract
Deep Neural Networks (DNNs) have shown great success in completing complex tasks. However, DNNs inevitably bring high computational cost and storage consumption due to the complexity of hierarchical structures, thereby hindering their wide deployment in Internet-of-Things (IoT) devices, which have limited computational capability and storage capacity. Therefore, it is a necessity to investigate the technologies to compact DNNs. Despite tremendous advances in compacting DNNs, few surveys summarize compacting-DNNs technologies, especially for IoT applications. Hence, this paper presents a comprehensive study on compacting-DNNs technologies. We categorize compacting-DNNs technologies into three major types: 1) network model compression, 2) Knowledge Distillation (KD), 3) modification of network structures. We also elaborate on the diversity of these approaches and make side-by-side comparisons. Moreover, we discuss the applications of compacted DNNs in various IoT applications and outline future directions.
arXiv comment25 pages, 11 figures
References in corpus (69)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Deep Learning in Neural Networks: An Overview
- Distilling the Knowledge in a Neural Network
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
- Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
- Squeeze-and-Excitation Networks
- Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
- FitNets: Hints for Thin Deep Nets
- Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
- Do Deep Nets Really Need to be Deep?
- Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
- Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
- Blockchain for Internet of Things: A Survey
- Deep Learning with Limited Numerical Precision
- In-Edge AI: Intelligentizing Mobile Edge Computing, Caching and Communication by Federated Learning
- Compressing Deep Convolutional Networks using Vector Quantization
- Ternary Weight Networks
- Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures
- Predicting Parameters in Deep Learning
- Pruning Filters for Efficient ConvNets
- Blockchain Technologies for the Internet of Things: Research Issues and Challenges
- SNIP: Single-shot Network Pruning based on Connection Sensitivity
- Speeding up Convolutional Neural Networks with Low Rank Expansions
- Born Again Neural Networks
- Convolutional neural networks with low-rank regularization
- Like What You Like: Knowledge Distill via Neuron Selectivity Transfer
- XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks
- SlimYOLOv3: Narrower, Faster and Better for Real-Time UAV Applications
- Single Path One-Shot Neural Architecture Search with Uniform Sampling
- EmergencyNet: Efficient Aerial Image Classification for Drone-Based Emergency Monitoring Using Atrous Convolutional Feature Fusion
- Loss-aware Binarization of Deep Networks
- Unifying distillation and privileged information
- Large scale distributed neural network training through online distillation
- Layer-wise training convolutional neural networks with smaller filters for human activity recognition using wearable sensors
- Dynamic Network Surgery for Efficient DNNs
- Big Data Analytics for Large Scale Wireless Networks: Challenges and Opportunities
- GhostNet: More Features from Cheap Operations
- Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications
- Moonshine: Distilling with Cheap Convolutions
- Contrastive Representation Distillation
- A Survey on Deep Neural Network Compression: Challenges, Overview, and Solutions
- Towards the Limit of Network Quantization
- Learning to Branch: Accelerating Resource Allocation in Wireless Networks
- Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neurons
- Automated Pruning for Deep Neural Network Compression
- ShiftCNN: Generalized Low-Precision Architecture for Inference of Convolutional Neural Networks
- DeepIoT: Compressing Deep Neural Network Structures for Sensing Systems with a Compressor-Critic Framework
- Graph-based Knowledge Distillation by Multi-head Attention Network
- Compressing RNNs for IoT devices by 15-38x using Kronecker Products
- Data-Driven Sparse Structure Selection for Deep Neural Networks
- Weightless: Lossy Weight Encoding For Deep Neural Network Compression
- Training Shallow and Thin Networks for Acceleration via Knowledge Distillation with Conditional Adversarial Networks
- Learning Accurate Low-Bit Deep Neural Networks with Stochastic Quantization
- Lightweight Residual Densely Connected Convolutional Neural Network
- Residual Knowledge Distillation
- Knowledge Flow: Improve Upon Your Teachers
- Structured Knowledge Distillation for Dense Prediction
- Knowledge distillation via adaptive instance normalization
- Knowledge Projection for Deep Neural Networks
- SCAN: A Scalable Neural Networks Framework Towards Compact and Efficient Models
- MEAL: Multi-Model Ensemble via Adversarial Learning
- SEP-Nets: Small and Effective Pattern Networks
- Filter Grafting for Deep Neural Networks
- Graph Representation Learning via Multi-task Knowledge Distillation
- LightPAFF: A Two-Stage Distillation Framework for Pre-training and Fine-tuning
- TanhExp: A Smooth Activation Function with High Convergence Speed for Lightweight Neural Networks
- Sparsely Aggregated Convolutional Networks
- RTN: Reparameterized Ternary Network