Understanding and Improving Convolutional Neural Networks via Concatenated Rectified Linear Units

Abstract

Recently, convolutional neural networks (CNNs) have been used as a powerful tool to solve many problems of machine learning and computer vision. In this paper, we aim to provide insight on the property of convolutional neural networks, as well as a generic method to improve the performance of many CNN architectures. Specifically, we first examine existing CNN models and observe an intriguing property that the filters in the lower layers form pairs (i.e., filters with opposite phase). Inspired by our observation, we propose a novel, simple yet effective activation scheme called concatenated ReLU (CRelu) and theoretically analyze its reconstruction property in CNNs. We integrate CRelu into several state-of-the-art CNN architectures and demonstrate improvement in their recognition performance on CIFAR-10/100 and ImageNet datasets with fewer trainable parameters. Our results suggest that better understanding of the properties of CNNs can lead to significant performance improvement with a simple modification.

ICML 2016

References in corpus (6)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Striving for Simplicity: The All Convolutional Net
• Training Very Deep Networks
• Stochastic Pooling for Regularization of Deep Convolutional Neural Networks
• Scalable Bayesian Optimization Using Deep Neural Networks
• Stacked What-Where Auto-encoders

Cited by in corpus (14)

• A Survey of Model Compression and Acceleration for Deep Neural Networks
• Large Scale Adversarial Representation Learning
• The Shattered Gradients Problem: If resnets are the answer, then what is the question?
• QReLU and m-QReLU: Two novel quantum activation functions to aid medical diagnostics
• PixelBNN: Augmenting the PixelCNN with batch normalization and the presentation of a fast architecture for retinal vessel segmentation
• All You Need is Beyond a Good Init: Exploring Better Solution for Training Extremely Deep Convolutional Neural Networks with Orthonormality and Modulation
• Doubly Convolutional Neural Networks
• Invertible DenseNets with Concatenated LipSwish
• The Advantage Regret-Matching Actor-Critic
• Neural Taylor Approximations: Convergence and Exploration in Rectifier Networks
• Planck Limits on Cosmic String Tension Using Machine Learning
• Data Efficient Stagewise Knowledge Distillation
• MOSQUITO-NET: A deep learning based CADx system for malaria diagnosis along with model interpretation using GradCam and class activation maps
• Why Convolutional Networks Learn Oriented Bandpass Filters: Theory and Empirical Support