Improved training of binary networks for human pose estimation and image recognition

Abstract

Big neural networks trained on large datasets have advanced the state-of-the-art for a large variety of challenging problems, improving performance by a large margin. However, under low memory and limited computational power constraints, the accuracy on the same problems drops considerable. In this paper, we propose a series of techniques that significantly improve the accuracy of binarized neural networks (i.e networks where both the features and the weights are binary). We evaluate the proposed improvements on two diverse tasks: fine-grained recognition (human pose estimation) and large-scale image recognition (ImageNet classification). Specifically, we introduce a series of novel methodological changes including: (a) more appropriate activation functions, (b) reverse-order initialization, (c) progressive quantization, and (d) network stacking and show that these additions improve existing state-of-the-art network binarization techniques, significantly. Additionally, for the first time, we also investigate the extent to which network binarization and knowledge distillation can be combined. When tested on the challenging MPII dataset, our method shows a performance improvement of more than 4% in absolute terms. Finally, we further validate our findings by applying the proposed techniques for large-scale object recognition on the Imagenet dataset, on which we report a reduction of error rate by 4%.

References in corpus (16)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Distilling the Knowledge in a Neural Network
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• Joint Training of a Convolutional Network and a Graphical Model for Human Pose Estimation
• Fixed Point Quantization of Deep Convolutional Networks
• Human pose estimation via Convolutional Part Heatmap Regression
• Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights
• Training deep neural networks with low precision multiplications
• Adversarial PoseNet: A Structure-aware Convolutional Network for Human Pose Estimation
• Hierarchical binary CNNs for landmark localization with limited resources
• Multi-Scale Structure-Aware Network for Human Pose Estimation

Cited by in corpus (15)

• Binary Neural Networks: A Survey
• Training Binary Neural Networks with Real-to-Binary Convolutions
• Latent Weights Do Not Exist: Rethinking Binarized Neural Network Optimization
• Forward and Backward Information Retention for Accurate Binary Neural Networks
• Self-Distribution Binary Neural Networks
• Binarizing by Classification: Is soft function really necessary?
• Improving Accuracy of Binary Neural Networks using Unbalanced Activation Distribution
• Distribution-sensitive Information Retention for Accurate Binary Neural Network
• Improving Binary Neural Networks through Fully Utilizing Latent Weights
• High-Capacity Expert Binary Networks
• Lightweight 3D Human Pose Estimation Network Training Using Teacher-Student Learning
• ReCU: Reviving the Dead Weights in Binary Neural Networks
• Exact Backpropagation in Binary Weighted Networks with Group Weight Transformations
• PokeBNN: A Binary Pursuit of Lightweight Accuracy
• FTBNN: Rethinking Non-linearity for 1-bit CNNs and Going Beyond