Deep Roots: Improving CNN Efficiency with Hierarchical Filter Groups

Abstract

We propose a new method for creating computationally efficient and compact convolutional neural networks (CNNs) using a novel sparse connection structure that resembles a tree root. This allows a significant reduction in computational cost and number of parameters compared to state-of-the-art deep CNNs, without compromising accuracy, by exploiting the sparsity of inter-layer filter dependencies. We validate our approach by using it to train more efficient variants of state-of-the-art CNN architectures, evaluated on the CIFAR10 and ILSVRC datasets. Our results show similar or higher accuracy than the baseline architectures with much less computation, as measured by CPU and GPU timings. For example, for ResNet 50, our model has 40% fewer parameters, 45% fewer floating point operations, and is 31% (12%) faster on a CPU (GPU). For the deeper ResNet 200 our model has 25% fewer floating point operations and 44% fewer parameters, while maintaining state-of-the-art accuracy. For GoogLeNet, our model has 7% fewer parameters and is 21% (16%) faster on a CPU (GPU).

Updated full version of paper, in full letter paper two-column paper. Includes many textual changes, updated CIFAR10 results, and new analysis of inter/intra-layer correlation

References in corpus (9)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Improving neural networks by preventing co-adaptation of feature detectors
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Deep Learning with Limited Numerical Precision
• One weird trick for parallelizing convolutional neural networks
• Compressing Neural Networks with the Hashing Trick
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• Spectral Representations for Convolutional Neural Networks

Cited by in corpus (40)

• ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks
• Aggregated Residual Transformations for Deep Neural Networks
• HANet: A Hierarchical Attention Network for Change Detection With Bitemporal Very-High-Resolution Remote Sensing Images
• ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design
• DualConv: Dual Convolutional Kernels for Lightweight Deep Neural Networks
• The Power of Sparsity in Convolutional Neural Networks
• Applications and Techniques for Fast Machine Learning in Science
• Scalable multimodal convolutional networks for brain tumour segmentation
• DCFNet: Deep Neural Network with Decomposed Convolutional Filters
• DensePoint: Learning Densely Contextual Representation for Efficient Point Cloud Processing
• Log-DenseNet: How to Sparsify a DenseNet
• Frequency-aware Discriminative Feature Learning Supervised by Single-Center Loss for Face Forgery Detection
• A Closer Look at Structured Pruning for Neural Network Compression
• NASA: Neural Articulated Shape Approximation
• Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions
• ANTNets: Mobile Convolutional Neural Networks for Resource Efficient Image Classification
• Two-level Group Convolution
• BlockSwap: Fisher-guided Block Substitution for Network Compression on a Budget
• My(o) Armband Leaks Passwords: An EMG and IMU Based Keylogging Side-Channel Attack
• All You Need is a Few Shifts: Designing Efficient Convolutional Neural Networks for Image Classification
• Deep Convolutional Decision Jungle for Image Classification
• Temporally Distributed Networks for Fast Video Semantic Segmentation
• SAWNet: A Spatially Aware Deep Neural Network for 3D Point Cloud Processing
• Making EfficientNet More Efficient: Exploring Batch-Independent Normalization, Group Convolutions and Reduced Resolution Training
• E2GC: Energy-efficient Group Convolution in Deep Neural Networks
• Building Efficient Deep Neural Networks with Unitary Group Convolutions
• A Pre-defined Sparse Kernel Based Convolution for Deep CNNs
• Fully Learnable Group Convolution for Acceleration of Deep Neural Networks
• EditIQ: Automated Cinematic Editing of Static Wide-Angle Videos via Dialogue Interpretation and Saliency Cues
• Compressing Neural Networks: Towards Determining the Optimal Layer-wise Decomposition
• Accelerating Sparse Approximate Matrix Multiplication on GPUs
• LiPar: A Lightweight Parallel Learning Model for Practical In-Vehicle Network Intrusion Detection
• On the Demystification of Knowledge Distillation: A Residual Network Perspective
• GroupBERT: Enhanced Transformer Architecture with Efficient Grouped Structures
• Structured Sparsity Inducing Adaptive Optimizers for Deep Learning
• Dual-side Sparse Tensor Core
• Self-grouping Convolutional Neural Networks
• Efficient Structured Pruning and Architecture Searching for Group Convolution
• Fine-grained Optimization of Deep Neural Networks
• Comb Convolution for Efficient Convolutional Architecture