Fast Walsh-Hadamard Transform and Smooth-Thresholding Based Binary Layers in Deep Neural Networks

Abstract

In this paper, we propose a novel layer based on fast Walsh-Hadamard transform (WHT) and smooth-thresholding to replace $1\times 1$ convolution layers in deep neural networks. In the WHT domain, we denoise the transform domain coefficients using the new smooth-thresholding non-linearity, a smoothed version of the well-known soft-thresholding operator. We also introduce a family of multiplication-free operators from the basic 2$\times$2 Hadamard transform to implement $3\times 3$ depthwise separable convolution layers. Using these two types of layers, we replace the bottleneck layers in MobileNet-V2 to reduce the network's number of parameters with a slight loss in accuracy. For example, by replacing the final third bottleneck layers, we reduce the number of parameters from 2.270M to 540K. This reduces the accuracy from 95.21\% to 92.98\% on the CIFAR-10 dataset. Our approach significantly improves the speed of data processing. The fast Walsh-Hadamard transform has a computational complexity of $O(m{\log }_{2}m)$. As a result, it is computationally more efficient than the $1\times 1$ convolution layer. The fast Walsh-Hadamard layer processes a tensor in ${\mathbb{R}}^{10\times 32\times 32\times 1024}$ about 2 times faster than $1\times 1$ convolution layer on NVIDIA Jetson Nano computer board.

arXiv commentThe paper (v1) has been accepted to CVPR 2021 BiVision Workshop. We notice the final Conv2D is also a 1x1 convolution layer so we update the result with changing the layer in v2. In v3, we update citation 37 because its authorship changes. In v4, we propose the improved version of smooth thresholding called "weighted smooth thresholding"

References in corpus (11)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• Binarized Neural Networks
• Compressing Neural Networks with the Hashing Trick
• Fast-SCNN: Fast Semantic Segmentation Network
• Training Binary Neural Networks with Real-to-Binary Convolutions
• Hierarchical binary CNNs for landmark localization with limited resources
• Larq Compute Engine: Design, Benchmark, and Deploy State-of-the-Art Binarized Neural Networks
• Energy Efficient Hadamard Neural Networks
• GradiVeQ: Vector Quantization for Bandwidth-Efficient Gradient Aggregation in Distributed CNN Training
• S2-BNN: Bridging the Gap Between Self-Supervised Real and 1-bit Neural Networks via Guided Distribution Calibration