Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

Abstract

Rectified activation units (rectifiers) are essential for state-of-the-art neural networks. In this work, we study rectifier neural networks for image classification from two aspects. First, we propose a Parametric Rectified Linear Unit (PReLU) that generalizes the traditional rectified unit. PReLU improves model fitting with nearly zero extra computational cost and little overfitting risk. Second, we derive a robust initialization method that particularly considers the rectifier nonlinearities. This method enables us to train extremely deep rectified models directly from scratch and to investigate deeper or wider network architectures. Based on our PReLU networks (PReLU-nets), we achieve 4.94% top-5 test error on the ImageNet 2012 classification dataset. This is a 26% relative improvement over the ILSVRC 2014 winner (GoogLeNet, 6.66%). To our knowledge, our result is the first to surpass human-level performance (5.1%, Russakovsky et al.) on this visual recognition challenge.

References in corpus (2)

• Return of the Devil in the Details: Delving Deep into Convolutional Nets
• Learning Activation Functions to Improve Deep Neural Networks

Cited by in corpus (163)

• Efficient Multi-Scale 3D CNN with Fully Connected CRF for Accurate Brain Lesion Segmentation
• Empirical Evaluation of Rectified Activations in Convolutional Network
• Deep Bilateral Learning for Real-Time Image Enhancement
• Deep Learning Microscopy
• Large-Scale Evolution of Image Classifiers
• Evolution of Image Segmentation using Deep Convolutional Neural Network: A Survey
• Systematic evaluation of CNN advances on the ImageNet
• Collaborative Memory Network for Recommendation Systems
• Learning Discrete Representations via Information Maximizing Self-Augmented Training
• Multiplicative Normalizing Flows for Variational Bayesian Neural Networks
• SD-CNN: a Shallow-Deep CNN for Improved Breast Cancer Diagnosis
• Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis
• A fast and accurate physics-informed neural network reduced order model with shallow masked autoencoder
• A Deep Learning Interpretable Classifier for Diabetic Retinopathy Disease Grading
• CMU DeepLens: Deep Learning For Automatic Image-based Galaxy-Galaxy Strong Lens Finding
• Momentum Contrastive Learning for Few-Shot COVID-19 Diagnosis from Chest CT Images
• The Strong Gravitational Lens Finding Challenge
• Super Resolution Convolutional Neural Network Models for Enhancing Resolution of Rock Micro-CT Images
• Sharp Minima Can Generalize For Deep Nets
• Searching for Exoplanets Using Artificial Intelligence
• Lagged correlation-based deep learning for directional trend change prediction in financial time series
• MgNet: A Unified Framework of Multigrid and Convolutional Neural Network
• Finding Strong Gravitational Lenses in the DESI DECam Legacy Survey
• The Shattered Gradients Problem: If resnets are the answer, then what is the question?
• Deep Learning with S-shaped Rectified Linear Activation Units
• Radio Galaxy Zoo: Compact and extended radio source classification with deep learning
• Exploiting deep residual networks for human action recognition from skeletal data
• A Deep Learning-based Reconstruction of Cosmic Ray-induced Air Showers
• Crowd counting with crowd attention convolutional neural network
• Learning Koopman Invariant Subspaces for Dynamic Mode Decomposition
• Deep learning enables reference-free isotropic super-resolution for volumetric fluorescence microscopy
• Machine-learning Approaches to Exoplanet Transit Detection and Candidate Validation in Wide-field Ground-based Surveys
• GSAE: an autoencoder with embedded gene-set nodes for genomics functional characterization
• A convolutional neural-network model of human cochlear mechanics and filter tuning for real-time applications
• A Genetic Programming Approach to Designing Convolutional Neural Network Architectures
• Multi-Representational Learning for Offline Signature Verification using Multi-Loss Snapshot Ensemble of CNNs
• Adversarial Multimodal Representation Learning for Click-Through Rate Prediction
• Deblending galaxy superpositions with branched generative adversarial networks
• Brain segmentation based on multi-atlas guided 3D fully convolutional network ensembles
• A Survey on Deep Learning Methods for Robot Vision
• Deep Learning for Computational Chemistry
• MobileFaceNets: Efficient CNNs for Accurate Real-Time Face Verification on Mobile Devices
• Equivariant Energy Flow Networks for Jet Tagging
• Machine learning astrophysics from 21 cm lightcones: impact of network architectures and signal contamination
• Graph Representation Learning via Graphical Mutual Information Maximization
• Understanding and Improving Fast Adversarial Training
• Deep Learning Convolutional Networks for Multiphoton Microscopy Vasculature Segmentation
• DECAF: Deep Extreme Classification with Label Features
• Impact of Artificial Intelligence on Businesses: from Research, Innovation, Market Deployment to Future Shifts in Business Models
• Deep Interest Network for Click-Through Rate Prediction
• A global method to identify trees outside of closed-canopy forests with medium-resolution satellite imagery
• NAIS-Net: Stable Deep Networks from Non-Autonomous Differential Equations
• Strong lensing in UNIONS: Toward a pipeline from discovery to modeling
• Deep learning jet modifications in heavy-ion collisions
• Molecule Identification with Rotational Spectroscopy and Probabilistic Deep Learning
• PointCleanNet: Learning to Denoise and Remove Outliers from Dense Point Clouds
• Geometric Uncertainty in Patient-Specific Cardiovascular Modeling with Convolutional Dropout Networks
• Event reconstruction for KM3NeT/ORCA using convolutional neural networks
• Global Extreme Heat Forecasting Using Neural Weather Models
• How Deep is the Feature Analysis underlying Rapid Visual Categorization?
• Shared Data and Algorithms for Deep Learning in Fundamental Physics
• Deep Learning For Face Recognition: A Critical Analysis
• Image Colorization Using a Deep Convolutional Neural Network
• EENMF: An End-to-End Neural Matching Framework for E-Commerce Sponsored Search
• Learning with hidden variables
• A neural network-based methodology to select young stellar object candidates from IR surveys
• Classification of 12-Lead ECG Signals with Bi-directional LSTM Network
• BIRNet: Brain Image Registration Using Dual-Supervised Fully Convolutional Networks
• GaborNet: Gabor filters with learnable parameters in deep convolutional neural networks
• Multi-Scale Deep Learning for Estimating Horizontal Velocity Fields on the Solar Surface
• Investigating the Impact of CNN Depth on Neonatal Seizure Detection Performance
• Exact and Consistent Interpretation for Piecewise Linear Neural Networks: A Closed Form Solution
• Deep Embedding Convolutional Neural Network for Synthesizing CT Image from T1-Weighted MR Image
• Fully Automatic Wound Segmentation with Deep Convolutional Neural Networks
• Identifying MgII Narrow Absorption Lines with Deep Learning
• Online and Offline Handwritten Chinese Character Recognition: A Comprehensive Study and New Benchmark
• Multi-Path Feedback Recurrent Neural Network for Scene Parsing
• E-swish: Adjusting Activations to Different Network Depths
• Application of Transfer Learning to Neutrino Interaction Classification
• On the impact of selected modern deep-learning techniques to the performance and celerity of classification models in an experimental high-energy physics use case
• Analyzing the Galactic pulsar distribution with machine learning
• Interacting Attention-gated Recurrent Networks for Recommendation
• Recurrent Localization Networks applied to the Lippmann-Schwinger Equation
• Taxonomy of Saliency Metrics for Channel Pruning
• Initialization Matters: Regularizing Manifold-informed Initialization for Neural Recommendation Systems
• Cascaded Context Pyramid for Full-Resolution 3D Semantic Scene Completion
• Learning to Respond with Your Favorite Stickers: A Framework of Unifying Multi-Modality and User Preference in Multi-Turn Dialog
• BoundarySqueeze: Image Segmentation as Boundary Squeezing
• A concatenating framework of shortcut convolutional neural networks
• Disentangled Representations in Neural Models
• Learning from Multi-domain Artistic Images for Arbitrary Style Transfer
• Automating Generation of Low Precision Deep Learning Operators
• Conditional Deep Learning for Energy-Efficient and Enhanced Pattern Recognition
• Deep learning insights into non-universality in the halo mass function
• MinCall - MinION end2end convolutional deep learning basecaller
• An Attention-Gated Convolutional Neural Network for Sentence Classification
• Trial by FIRE: Probing the dark matter density profile of dwarf galaxies with GraphNPE
• QuickNet: Maximizing Efficiency and Efficacy in Deep Architectures
• Accelerating multijet-merged event generation with neural network matrix element surrogates
• Lean CNNs for mapping electron charge density fields to material properties
• C3VQG: Category Consistent Cyclic Visual Question Generation
• A 1d convolutional network for leaf and time series classification
• Robust and On-the-fly Dataset Denoising for Image Classification
• Learning Context-Dependent Choice Functions
• Accurate Automatic Segmentation of Amygdala Subnuclei and Modeling of Uncertainty via Bayesian Fully Convolutional Neural Network
• Deep Online Convex Optimization with Gated Games
• Multi-Source Fusion and Automatic Predictor Selection for Zero-Shot Video Object Segmentation
• Dual-energy CT Reconstruction from Dual Quarter Scans
• Classifying CMB time-ordered data through deep neural networks
• On Deep Representation Learning from Noisy Web Images
• MAGIC: Microlensing Analysis Guided by Intelligent Computation
• Unsupervised Regenerative Learning of Hierarchical Features in Spiking Deep Networks for Object Recognition
• An Adversarial Transfer Network for Knowledge Representation Learning
• The Evolution of Neural Network-Based Chart Patterns: A Preliminary Study
• Learning to Respond with Stickers: A Framework of Unifying Multi-Modality in Multi-Turn Dialog
• Fast, Better Training Trick -- Random Gradient
• MIASSR: An Approach for Medical Image Arbitrary Scale Super-Resolution
• Mitigating Imperfections in Mixed-Signal Neuromorphic Circuits
• A deep neural network to enhance prediction of 1-year mortality using echocardiographic videos of the heart
• Physics-Informed Neural Networks for Solving Multiscale Mode-Resolved Phonon Boltzmann Transport Equation
• Mean Shift Rejection: Training Deep Neural Networks Without Minibatch Statistics or Normalization
• Reconstruction of Simulation-Based Physical Field by Reconstruction Neural Network Method
• Fast, invariant representation for human action in the visual system
• Learn-able parameter guided Activation Functions
• Segmentation of optic disc, fovea and retinal vasculature using a single convolutional neural network
• Multi-Task Deep Learning for Legal Document Translation, Summarization and Multi-Label Classification
• Unitary Learning for Deep Diffractive Neural Network
• Exploiting the Value of the Center-dark Channel Prior for Salient Object Detection
• Multiple receptive fields and small-object-focusing weakly-supervised segmentation network for fast object detection
• Self-Supervised Learning from Unlabeled Fundus Photographs Improves Segmentation of the Retina
• Comparing the costs of abstraction for DL frameworks
• Deep Octonion Networks
• Soft Autoencoder and Its Wavelet Adaptation Interpretation
• Overcoming Overfitting and Large Weight Update Problem in Linear Rectifiers: Thresholded Exponential Rectified Linear Units
• Image Smoothing via Unsupervised Learning
• A Miniaturized Semantic Segmentation Method for Remote Sensing Image
• The neural and cognitive architecture for learning from a small sample
• Adaptive Regularization via Residual Smoothing in Deep Learning Optimization
• Learning Boolean functions with concentrated spectra
• Semi-Supervised Graph Classification: A Hierarchical Graph Perspective
• A Method for Restoring the Training Set Distribution in an Image Classifier
• Dense Regression Activation Maps For Lesion Segmentation in CT scans of COVID-19 patients
• Learning Image Matching by Simply Watching Video
• Lenses In VoicE (LIVE): Searching for strong gravitational lenses in the VOICE@VST survey using Convolutional Neural Networks
• Tensorization of neural networks for improved privacy and interpretability
• Differentiable Disentanglement Filter: an Application Agnostic Core Concept Discovery Probe
• Convergence Analysis of the Dynamics of a Special Kind of Two-Layered Neural Networks with ${\ell }_1$ and ${\ell }_2$ Regularization
• Anchor-based Plain Net for Mobile Image Super-Resolution
• Predicting the success of Gradient Descent for a particular Dataset-Architecture-Initialization (DAI)
• Global Structure-Aware Drum Transcription Based on Self-Attention Mechanisms
• Scale Normalization
• Predicting Path Failure In Time-Evolving Graphs
• Automatic deep learning for trend prediction in time series data
• What and Where to Translate: Local Mask-based Image-to-Image Translation
• MultiHead MultiModal Deep Interest Recommendation Network
• A Quadratic Actor Network for Model-Free Reinforcement Learning
• Residual-CNDS for Grand Challenge Scene Dataset
• Deep CTR Prediction in Display Advertising
• Machine Learning based Post Processing Artifact Reduction in HEVC Intra Coding
• Mining Domain Knowledge: Improved Framework towards Automatically Standardizing Anatomical Structure Nomenclature in Radiotherapy
• On the Selective and Invariant Representation of DCNN for High-Resolution Remote Sensing Image Recognition
• The Compressed Model of Residual CNDS
• Deep Control - a simple automatic gain control for memory efficient and high performance training of deep convolutional neural networks