Multi-Bias Non-linear Activation in Deep Neural Networks

Abstract

As a widely used non-linear activation, Rectified Linear Unit (ReLU) separates noise and signal in a feature map by learning a threshold or bias. However, we argue that the classification of noise and signal not only depends on the magnitude of responses, but also the context of how the feature responses would be used to detect more abstract patterns in higher layers. In order to output multiple response maps with magnitude in different ranges for a particular visual pattern, existing networks employing ReLU and its variants have to learn a large number of redundant filters. In this paper, we propose a multi-bias non-linear activation (MBA) layer to explore the information hidden in the magnitudes of responses. It is placed after the convolution layer to decouple the responses to a convolution kernel into multiple maps by multi-thresholding magnitudes, thus generating more patterns in the feature space at a low computational cost. It provides great flexibility of selecting responses to different visual patterns in different magnitude ranges to form rich representations in higher layers. Such a simple and yet effective scheme achieves the state-of-the-art performance on several benchmarks.

References in corpus (2)

• Stochastic Pooling for Regularization of Deep Convolutional Neural Networks
• Scalable Bayesian Optimization Using Deep Neural Networks

Cited by in corpus (8)

• A Survey of Model Compression and Acceleration for Deep Neural Networks
• Systematic evaluation of CNN advances on the ImageNet
• Dual Deep Network for Visual Tracking
• Rethinking Feature Discrimination and Polymerization for Large-scale Recognition
• Regularizing CNNs with Locally Constrained Decorrelations
• Zoom Out-and-In Network with Recursive Training for Object Proposal
• Crafting GBD-Net for Object Detection
• Multi-bin Trainable Linear Unit for Fast Image Restoration Networks