Batch Kalman Normalization: Towards Training Deep Neural Networks with Micro-Batches

Abstract

As an indispensable component, Batch Normalization (BN) has successfully improved the training of deep neural networks (DNNs) with mini-batches, by normalizing the distribution of the internal representation for each hidden layer. However, the effectiveness of BN would diminish with scenario of micro-batch (e.g., less than 10 samples in a mini-batch), since the estimated statistics in a mini-batch are not reliable with insufficient samples. In this paper, we present a novel normalization method, called Batch Kalman Normalization (BKN), for improving and accelerating the training of DNNs, particularly under the context of micro-batches. Specifically, unlike the existing solutions treating each hidden layer as an isolated system, BKN treats all the layers in a network as a whole system, and estimates the statistics of a certain layer by considering the distributions of all its preceding layers, mimicking the merits of Kalman Filtering. BKN has two appealing properties. First, it enables more stable training and faster convergence compared to previous works. Second, training DNNs using BKN performs substantially better than those using BN and its variants, especially when very small mini-batches are presented. On the image classification benchmark of ImageNet, using BKN powered networks we improve upon the best-published model-zoo results: reaching 74.0% top-1 val accuracy for InceptionV2. More importantly, using BKN achieves the comparable accuracy with extremely smaller batch size, such as 64 times smaller on CIFAR-10/100 and 8 times smaller on ImageNet.

arXiv commentWe presented how to improve and accelerate the training of DNNs, particularly under the context of micro-batches. (Submitted to IJCAI 2018)

References in corpus (8)

• Deep Residual Learning for Image Recognition
• Going Deeper with Convolutions
• Improved Techniques for Training GANs
• Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
• Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models
• Normalization Propagation: A Parametric Technique for Removing Internal Covariate Shift in Deep Networks
• Natural Neural Networks
• Recurrent Batch Normalization

Cited by in corpus (8)

• Differentiable Learning-to-Normalize via Switchable Normalization
• Switchable Whitening for Deep Representation Learning
• Adaptively Connected Neural Networks
• Kalman meets Bellman: Improving Policy Evaluation through Value Tracking
• Trust Region Value Optimization using Kalman Filtering
• TRADI: Tracking deep neural network weight distributions for uncertainty estimation
• U-Net Training with Instance-Layer Normalization
• Batch Normalization Sampling