Dense xUnit Networks

Abstract

Deep net architectures have constantly evolved over the past few years, leading to significant advancements in a wide array of computer vision tasks. However, besides high accuracy, many applications also require a low computational load and limited memory footprint. To date, efficiency has typically been achieved either by architectural choices at the macro level (e.g. using skip connections or pruning techniques) or modifications at the level of the individual layers (e.g. using depth-wise convolutions or channel shuffle operations). Interestingly, much less attention has been devoted to the role of the activation functions in constructing efficient nets. Recently, Kligvasser et al. showed that incorporating spatial connections within the activation functions, enables a significant boost in performance in image restoration tasks, at any given budget of parameters. However, the effectiveness of their xUnit module has only been tested on simple small models, which are not characteristic of those used in high-level vision tasks. In this paper, we adopt and improve the xUnit activation, show how it can be incorporated into the DenseNet architecture, and illustrate its high effectiveness for classification and image restoration tasks alike. While the DenseNet architecture is extremely efficient to begin with, our dense xUnit net (DxNet) can typically achieve the same performance with far fewer parameters. For example, on ImageNet, our DxNet outperforms a ReLU-based DenseNet having 30% more parameters and achieves state-of-the-art results for this budget of parameters. Furthermore, in denoising and super-resolution, DxNet significantly improves upon all existing lightweight solutions, including the xUnit-based nets of Kligvasser et al.

References in corpus (19)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising
• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• Deep Residual Learning for Image Recognition
• Improved Regularization of Convolutional Neural Networks with Cutout
• MobileNetV2: Inverted Residuals and Linear Bottlenecks
• Densely Connected Convolutional Networks
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• Binarized Neural Networks
• Trained Ternary Quantization
• Gather-Excite: Exploiting Feature Context in Convolutional Neural Networks
• YOLO9000: Better, Faster, Stronger
• Xception: Deep Learning with Depthwise Separable Convolutions
• Learning Deep Features for Discriminative Localization
• MemNet: A Persistent Memory Network for Image Restoration
• Learning Deep CNN Denoiser Prior for Image Restoration

Cited by in corpus (1)

• Efficient Human Pose Estimation by Learning Deeply Aggregated Representations