Deep Residual Networks with Exponential Linear Unit

Abstract

Very deep convolutional neural networks introduced new problems like vanishing gradient and degradation. The recent successful contributions towards solving these problems are Residual and Highway Networks. These networks introduce skip connections that allow the information (from the input or those learned in earlier layers) to flow more into the deeper layers. These very deep models have lead to a considerable decrease in test errors, on benchmarks like ImageNet and COCO. In this paper, we propose the use of exponential linear unit instead of the combination of ReLU and Batch Normalization in Residual Networks. We show that this not only speeds up learning in Residual Networks but also improves the accuracy as the depth increases. It improves the test error on almost all data sets, like CIFAR-10 and CIFAR-100

arXiv commentsubmitted in Vision Net 2016, Jaipur, India

References in corpus (12)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Striving for Simplicity: The All Convolutional Net
• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
• Training Very Deep Networks
• OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks
• Deeply-Supervised Nets
• Highway Networks
• Scalable Bayesian Optimization Using Deep Neural Networks
• Learning Activation Functions to Improve Deep Neural Networks
• Fractional Max-Pooling
• Generalizing Pooling Functions in Convolutional Neural Networks: Mixed, Gated, and Tree

Cited by in corpus (20)

• Gaussian Error Linear Units (GELUs)
• A survey on modern trainable activation functions
• Residual Networks of Residual Networks: Multilevel Residual Networks
• A Geometric Analysis of Neural Collapse with Unconstrained Features
• Acoustic Scene Classification with Squeeze-Excitation Residual Networks
• Individual Recognition in Schizophrenia using Deep Learning Methods with Random Forest and Voting Classifiers: Insights from Resting State EEG Streams
• Reconstructing undersampled photoacoustic microscopy images using deep learning
• Modelling the galaxy-halo connection with semi-recurrent neural networks
• Genetic Architect: Discovering Genomic Structure with Learned Neural Architectures
• Recent Advances in the Applications of Convolutional Neural Networks to Medical Image Contour Detection
• Improving Deep Neural Network with Multiple Parametric Exponential Linear Units
• Spherical View Synthesis for Self-Supervised 360 Depth Estimation
• Deep Artifact-Free Residual Network for Single Image Super-Resolution
• MinCall - MinION end2end convolutional deep learning basecaller
• Compressing Deep Neural Networks via Layer Fusion
• UPANets: Learning from the Universal Pixel Attention Networks
• DelugeNets: Deep Networks with Efficient and Flexible Cross-layer Information Inflows
• BA^2M: A Batch Aware Attention Module for Image Classification
• Grading video interviews with fairness considerations
• CRL: Class Representative Learning for Image Classification