Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

Abstract

Rectified activation units (rectifiers) are essential for state-of-the-art neural networks. In this work, we study rectifier neural networks for image classification from two aspects. First, we propose a Parametric Rectified Linear Unit (PReLU) that generalizes the traditional rectified unit. PReLU improves model fitting with nearly zero extra computational cost and little overfitting risk. Second, we derive a robust initialization method that particularly considers the rectifier nonlinearities. This method enables us to train extremely deep rectified models directly from scratch and to investigate deeper or wider network architectures. Based on our PReLU networks (PReLU-nets), we achieve 4.94% top-5 test error on the ImageNet 2012 classification dataset. This is a 26% relative improvement over the ILSVRC 2014 winner (GoogLeNet, 6.66%). To our knowledge, our result is the first to surpass human-level performance (5.1%, Russakovsky et al.) on this visual recognition challenge.

References in corpus (2)

• Return of the Devil in the Details: Delving Deep into Convolutional Nets
• Learning Activation Functions to Improve Deep Neural Networks

Cited by in corpus (29)

• Deep Bilateral Learning for Real-Time Image Enhancement
• Deep Learning Microscopy
• Super Resolution Convolutional Neural Network Models for Enhancing Resolution of Rock Micro-CT Images
• Searching for Exoplanets Using Artificial Intelligence
• A Deep Learning-based Reconstruction of Cosmic Ray-induced Air Showers
• Learning Koopman Invariant Subspaces for Dynamic Mode Decomposition
• Impact of Artificial Intelligence on Businesses: from Research, Innovation, Market Deployment to Future Shifts in Business Models
• GaborNet: Gabor filters with learnable parameters in deep convolutional neural networks
• Deep Embedding Convolutional Neural Network for Synthesizing CT Image from T1-Weighted MR Image
• Identifying MgII Narrow Absorption Lines with Deep Learning
• E-swish: Adjusting Activations to Different Network Depths
• Multi-Path Feedback Recurrent Neural Network for Scene Parsing
• Interacting Attention-gated Recurrent Networks for Recommendation
• A concatenating framework of shortcut convolutional neural networks
• MinCall - MinION end2end convolutional deep learning basecaller
• Accurate Automatic Segmentation of Amygdala Subnuclei and Modeling of Uncertainty via Bayesian Fully Convolutional Neural Network
• QuickNet: Maximizing Efficiency and Efficacy in Deep Architectures
• The Evolution of Neural Network-Based Chart Patterns: A Preliminary Study
• Segmentation of optic disc, fovea and retinal vasculature using a single convolutional neural network
• Multiple receptive fields and small-object-focusing weakly-supervised segmentation network for fast object detection
• Deep Octonion Networks
• Semi-Supervised Graph Classification: A Hierarchical Graph Perspective
• Deep Control - a simple automatic gain control for memory efficient and high performance training of deep convolutional neural networks
• The Compressed Model of Residual CNDS
• On the Selective and Invariant Representation of DCNN for High-Resolution Remote Sensing Image Recognition
• Convergence Analysis of the Dynamics of a Special Kind of Two-Layered Neural Networks with ${\ell }_1$ and ${\ell }_2$ Regularization
• Predicting Path Failure In Time-Evolving Graphs
• Deep CTR Prediction in Display Advertising
• Differentiable Disentanglement Filter: an Application Agnostic Core Concept Discovery Probe