Batch Normalization with Enhanced Linear Transformation
arXiv:2011.14150
Abstract
Batch normalization (BN) is a fundamental unit in modern deep networks, in which a linear transformation module was designed for improving BN's flexibility of fitting complex data distributions. In this paper, we demonstrate properly enhancing this linear transformation module can effectively improve the ability of BN. Specifically, rather than using a single neuron, we propose to additionally consider each neuron's neighborhood for calculating the outputs of the linear transformation. Our method, named BNET, can be implemented with 2-3 lines of code in most deep learning libraries. Despite the simplicity, BNET brings consistent performance gains over a wide range of backbones and visual benchmarks. Moreover, we verify that BNET accelerates the convergence of network training and enhances spatial information by assigning the important neurons with larger weights accordingly. The code is available at https://github.com/yuhuixu1993/BNET.
12 pages. The code is available at https://github.com/yuhuixu1993/BNET
References in corpus (6)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Rethinking Atrous Convolution for Semantic Image Segmentation
- UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
- MMDetection: Open MMLab Detection Toolbox and Benchmark
- Deep Isometric Learning for Visual Recognition