CondenseNet: An Efficient DenseNet using Learned Group Convolutions

Abstract

Deep neural networks are increasingly used on mobile devices, where computational resources are limited. In this paper we develop CondenseNet, a novel network architecture with unprecedented efficiency. It combines dense connectivity with a novel module called learned group convolution. The dense connectivity facilitates feature re-use in the network, whereas learned group convolutions remove connections between layers for which this feature re-use is superfluous. At test time, our model can be implemented using standard group convolutions, allowing for efficient computation in practice. Our experiments show that CondenseNets are far more efficient than state-of-the-art compact convolutional networks such as MobileNets and ShuffleNets.

References in corpus (9)

• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Striving for Simplicity: The All Convolutional Net
• FitNets: Hints for Thin Deep Nets
• Going Deeper with Convolutions
• Pruning Filters for Efficient ConvNets
• Compressing Neural Networks with the Hashing Trick
• Channel Pruning for Accelerating Very Deep Neural Networks
• Interleaved Group Convolutions for Deep Neural Networks

Cited by in corpus (41)

• Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting
• Deep Learning Based Text Classification: A Comprehensive Review
• Knowledge Distillation by On-the-Fly Native Ensemble
• Big Bird: Transformers for Longer Sequences
• ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design
• Path-Level Network Transformation for Efficient Architecture Search
• MONAS: Multi-Objective Neural Architecture Search using Reinforcement Learning
• Benchmarking Neural Network Robustness to Common Corruptions and Surface Variations
• Packing Sparse Convolutional Neural Networks for Efficient Systolic Array Implementations: Column Combining Under Joint Optimization
• A Programmable Approach to Neural Network Compression
• DPP-Net: Device-aware Progressive Search for Pareto-optimal Neural Architectures
• ChamNet: Towards Efficient Network Design through Platform-Aware Model Adaptation
• Blockwise Self-Attention for Long Document Understanding
• Selfish Sparse RNN Training
• Anytime Inference with Distilled Hierarchical Neural Ensembles
• The Power of Selecting Key Blocks with Local Pre-ranking for Long Document Information Retrieval
• Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Masks
• TT-Rec: Tensor Train Compression for Deep Learning Recommendation Models
• Efficient Forward Architecture Search
• Scatterbrain: Unifying Sparse and Low-rank Attention Approximation
• S2RMs: Spatially Structured Recurrent Modules
• SparseRT: Accelerating Unstructured Sparsity on GPUs for Deep Learning Inference
• HSD-CNN: Hierarchically self decomposing CNN architecture using class specific filter sensitivity analysis
• Term Revealing: Furthering Quantization at Run Time on Quantized DNNs
• Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Models
• Training Deep Neural Networks with Joint Quantization and Pruning of Weights and Activations
• AOGNets: Compositional Grammatical Architectures for Deep Learning
• Set-to-Sequence Methods in Machine Learning: a Review
• Accelerating SpMM Kernel with Cache-First Edge Sampling for Graph Neural Networks
• FlexSA: Flexible Systolic Array Architecture for Efficient Pruned DNN Model Training
• Low-Power Computer Vision: Status, Challenges, Opportunities
• Stochastic Gradient Descent with Hyperbolic-Tangent Decay on Classification
• SparseDNN: Fast Sparse Deep Learning Inference on CPUs
• ELASTIC: Improving CNNs with Dynamic Scaling Policies
• DECORE: Deep Compression with Reinforcement Learning
• Efficient Semantic Segmentation using Gradual Grouping
• Accelerated CNN Training Through Gradient Approximation
• Neural Networks at a Fraction with Pruned Quaternions
• ESPN: Extremely Sparse Pruned Networks
• Semantically Selective Augmentation for Deep Compact Person Re-Identification
• Full-stack Optimization for Accelerating CNNs with FPGA Validation