Batch Normalization with Enhanced Linear Transformation
arXiv:2011.14150
Abstract
Batch normalization (BN) is a fundamental unit in modern deep networks, in which a linear transformation module was designed for improving BN's flexibility of fitting complex data distributions. In this paper, we demonstrate properly enhancing this linear transformation module can effectively improve the ability of BN. Specifically, rather than using a single neuron, we propose to additionally consider each neuron's neighborhood for calculating the outputs of the linear transformation. Our method, named BNET, can be implemented with 2-3 lines of code in most deep learning libraries. Despite the simplicity, BNET brings consistent performance gains over a wide range of backbones and visual benchmarks. Moreover, we verify that BNET accelerates the convergence of network training and enhances spatial information by assigning the important neurons with larger weights accordingly. The code is available at https://github.com/yuhuixu1993/BNET.
arXiv comment12 pages. The code is available at https://github.com/yuhuixu1993/BNET
References in corpus (9)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Rethinking Atrous Convolution for Semantic Image Segmentation
- UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
- Instance Normalization: The Missing Ingredient for Fast Stylization
- MMDetection: Open MMLab Detection Toolbox and Benchmark
- Micro-Batch Training with Batch-Channel Normalization and Weight Standardization
- Deep Isometric Learning for Visual Recognition