Deep Learning using Rectified Linear Units (ReLU)

Abstract

The Rectified Linear Unit (ReLU) is a foundational activation function in artficial neural networks. Recent literature frequently misattributes its origin to the 2018 (initial) version of this paper, which exclusively investigated ReLU at the classification layer. This paper formally corrects the citation record by tracing the mathematical lineage of piecewise linear functions from early biological models to their definitive integration into deep learning by Nair & Hinton (2010). Alongside this historical rectification, we present a comprehensive empirical comparison of the ReLU, Hyperbolic Tangent (Tanh), and Logistic (Sigmoid) activation functions across image classification, text classification, and image reconstruction tasks. To ensure statistical robustness, we evaluated these functions using 10 independent randomized trials and assessed significance using the non-parametric Kruskal-Wallis $H$ test. The empirical data validates the theoretical limitations of saturating functions. Sigmoid failed to converge in deep convolutional vision tasks due to the vanishing gradient problem, thus yielding accuracies equivalent to random probability. Conversely, ReLU and Tanh exhibited stable convergence. ReLU achieved the highest mean accuracy and F1-score on image classification and text classification tasks, while Tanh yielded the highest peak signal to noise ratio in image reconstruction. Ultimately, this study confirms a statistically significant performance variance among activations, thus reaffirming the necessity of non-saturating functions in deep architectures, and restores proper historical attribution to prior literature.

9 pages, 5 figures, 5 tables

References in corpus (7)

• Adam: A Method for Stochastic Optimization
• Scikit-learn: Machine Learning in Python
• The NumPy array: a structure for efficient numerical computation
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Attention-Based Models for Speech Recognition
• Deep Learning using Linear Support Vector Machines
• A Neural Network Architecture Combining Gated Recurrent Unit (GRU) and Support Vector Machine (SVM) for Intrusion Detection in Network Traffic Data

Cited by in corpus (174)

• A Physics-Informed Machine Learning Approach for Solving Heat Transfer Equation in Advanced Manufacturing and Engineering Applications
• Hyper-Parameter Optimization: A Review of Algorithms and Applications
• Evolution of Image Segmentation using Deep Convolutional Neural Network: A Survey
• Quantum machine learning for image classification
• Recent Advances and Applications of Machine Learning in Experimental Solid Mechanics: A Review
• How Deep Learning Sees the World: A Survey on Adversarial Attacks & Defenses
• Decision-Focused Learning: Foundations, State of the Art, Benchmark and Future Opportunities
• CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval
• Weed Density and Distribution Estimation for Precision Agriculture using Semi-Supervised Learning
• Q-Learning based system for path planning with unmanned aerial vehicles swarms in obstacle environments
• Quantum materials for energy-efficient neuromorphic computing
• BanFakeNews: A Dataset for Detecting Fake News in Bangla
• Interaction networks for the identification of boosted $H\to b\overline{b}$ decays
• Boosting Fast Adversarial Training with Learnable Adversarial Initialization
• Convolutional Neural Network Ensemble Learning for Hyperspectral Imaging-based Blackberry Fruit Ripeness Detection in Uncontrolled Farm Environment
• iPromoter-BnCNN: a Novel Branched CNN Based Predictor for Identifying and Classifying Sigma Promoters
• KPGT: Knowledge-Guided Pre-training of Graph Transformer for Molecular Property Prediction
• Hyperspectral Image Classification With Contrastive Graph Convolutional Network
• Flakify: A Black-Box, Language Model-based Predictor for Flaky Tests
• On Deep Learning Techniques to Boost Monocular Depth Estimation for Autonomous Navigation
• Accelerating Reinforcement Learning for Reaching using Continuous Curriculum Learning
• Continuous Sign Language Recognition System using Deep Learning with MediaPipe Holistic
• Data-driven dose calculation algorithm based on deep learning
• Deep convolutional forest: a dynamic deep ensemble approach for spam detection in text
• Vertex and Energy Reconstruction in JUNO with Machine Learning Methods
• Wireless Charging Power Side-Channel Attacks
• Proactive Autoscaling for Edge Computing Systems with Kubernetes
• Reducing the complexity of chemical networks via interpretable autoencoders
• Contrastive Graph Convolutional Networks for Hardware Trojan Detection in Third Party IP Cores
• Detecting gravitational lenses using machine learning: exploring interpretability and sensitivity to rare lensing configurations
• Thermal experiments for fractured rock characterization: theoretical analysis and inverse modeling
• Semantic and Relational Spaces in Science of Science: Deep Learning Models for Article Vectorisation
• Cramér-Rao bound-informed training of neural networks for quantitative MRI
• QMaxViT-Unet+: A Query-Based MaxViT-Unet with Edge Enhancement for Scribble-Supervised Segmentation of Medical Images
• Well Googled is Half Done: Multimodal Forecasting of New Fashion Product Sales with Image-based Google Trends
• Galaxy classification: a deep learning approach for classifying Sloan Digital Sky Survey images
• An Empirical Study on Predictability of Software Code Smell Using Deep Learning Models
• DuETA: Traffic Congestion Propagation Pattern Modeling via Efficient Graph Learning for ETA Prediction at Baidu Maps
• RetroGraph: Retrosynthetic Planning with Graph Search
• GNN-Geo: A Graph Neural Network-based Fine-grained IP geolocation Framework
• FPGAs-as-a-Service Toolkit (FaaST)
• Dynamic Cone-beam CT Reconstruction using Spatial and Temporal Implicit Neural Representation Learning (STINR)
• Deriving star cluster parameters with convolutional neural networks. I. Age, mass, and size
• M2TRec: Metadata-aware Multi-task Transformer for Large-scale and Cold-start free Session-based Recommendations
• MeerCRAB: MeerLICHT Classification of Real and Bogus Transients using Deep Learning
• DPDnet: A Robust People Detector using Deep Learning with an Overhead Depth Camera
• Identification of point sources in gamma rays using U-shaped convolutional neural networks and a data challenge
• HRPVT: High-Resolution Pyramid Vision Transformer for medium and small-scale human pose estimation
• A novel method for error analysis in radiation thermometry with application to industrial furnaces
• Acoustic Leak Detection in Water Networks
• 1-Dimensional polynomial neural networks for audio signal related problems
• Modeling the sea-surface $p$CO${}_2$ of the central Bay of Bengal region using machine learning algorithms
• Dynamic imaging using Motion-Compensated SmooThness Regularization on Manifolds (MoCo-SToRM)
• Machine Learning Statistical Gravity from Multi-Region Entanglement Entropy
• SINR: Deconvolving Circular SAS Images Using Implicit Neural Representations
• A Cascade Transformer-based Model for 3D Dose Distribution Prediction in Head and Neck Cancer Radiotherapy
• Matrix Element Regression with Deep Neural Networks -- breaking the CPU barrier
• Highly-scalable, physics-informed GANs for learning solutions of stochastic PDEs
• DRL-Assisted Resource Allocation for NOMA-MEC Offloading with Hybrid SIC
• OnDev-LCT: On-Device Lightweight Convolutional Transformers towards federated learning
• RLAD: Reinforcement Learning from Pixels for Autonomous Driving in Urban Environments
• Explaining a Deep Reinforcement Learning Docking Agent Using Linear Model Trees with User Adapted Visualization
• Galaxy clustering from the bottom up: A Streaming Model emulator I
• GEDIT: Geographic-Enhanced and Dependency-Guided Tagging for Joint POI and Accessibility Extraction at Baidu Maps
• Vision-Aided Radio: User Identity Match in Radio and Video Domains Using Machine Learning
• TargetCall: Eliminating the Wasted Computation in Basecalling via Pre-Basecalling Filtering
• COVID-19 Diagnosis from Cough Acoustics using ConvNets and Data Augmentation
• Zen-NAS: A Zero-Shot NAS for High-Performance Deep Image Recognition
• Quantum Approximate Optimization Algorithm Parameter Prediction Using a Convolutional Neural Network
• The effect of Target Normalization and Momentum on Dying ReLU
• Prediction Model for Mortality Analysis of Pregnant Women Affected With COVID-19
• Painting baryons onto N-body simulations of galaxy clusters with image-to-image deep learning
• CapillaryNet: An Automated System to Quantify Skin Capillary Density and Red Blood Cell Velocity from Handheld Vital Microscopy
• iBA: Backdoor Attack on 3D Point Cloud via Reconstructing Itself
• Detection of Einstein Telescope gravitational wave signals from binary black holes using deep learning
• Empirical modeling and hybrid machine learning framework for nucleate pool boiling on microchannel structured surfaces
• One-dimensional Convolutional Neural Networks for Detecting Transiting Exoplanets
• CoRe Optimizer: An All-in-One Solution for Machine Learning
• Lifelong Person Re-Identification via Adaptive Knowledge Accumulation
• ST3D: Self-training for Unsupervised Domain Adaptation on 3D Object Detection
• Relation Aware Semi-autoregressive Semantic Parsing for NL2SQL
• Facial Expression Video Generation Based-On Spatio-temporal Convolutional GAN: FEV-GAN
• ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learning
• Causal-BALD: Deep Bayesian Active Learning of Outcomes to Infer Treatment-Effects from Observational Data
• Residual-Concatenate Neural Network with Deep Regularization Layers for Binary Classification
• A Visualized Malware Detection Framework with CNN and Conditional GAN
• HyperInverter: Improving StyleGAN Inversion via Hypernetwork
• Denoised Internal Models: a Brain-Inspired Autoencoder against Adversarial Attacks
• Machine learning classification: case of Higgs boson CP state in H to tau tau decay at LHC
• Search for emerging jets in $pp$ collisions at $\sqrt{s}=13.6$ TeV with the ATLAS experiment
• Feature Boosting, Suppression, and Diversification for Fine-Grained Visual Classification
• SHAMSUL: Systematic Holistic Analysis to investigate Medical Significance Utilizing Local interpretability methods in deep learning for chest radiography pathology prediction
• Capturing Financial markets to apply Deep Reinforcement Learning
• GradSign: Model Performance Inference with Theoretical Insights
• Audio2Gestures: Generating Diverse Gestures from Speech Audio with Conditional Variational Autoencoders
• Forged Image Detection using SOTA Image Classification Deep Learning Methods for Image Forensics with Error Level Analysis
• RLSS: A Deep Reinforcement Learning Algorithm for Sequential Scene Generation
• Sub-Goal Trees -- a Framework for Goal-Directed Trajectory Prediction and Optimization
• Growing Cosine Unit: A Novel Oscillatory Activation Function That Can Speedup Training and Reduce Parameters in Convolutional Neural Networks
• AGNet: Weighing Black Holes with Deep Learning
• Residual Matrix Product State for Machine Learning
• Real-time Artificial Intelligence for Accelerator Control: A Study at the Fermilab Booster
• Atss-Net: Target Speaker Separation via Attention-based Neural Network
• Detecting Thermodynamic Phase Transition via Explainable Machine Learning of Photoemission Spectroscopy
• Deep Residual Neural Networks for Image in Speech Steganography
• Self-Supervised Contrastive Learning for Robust Audio-Sheet Music Retrieval Systems
• Machine Learning Percolation Model
• Deep Learning for Power System Security Assessment
• Constraint Solving with Deep Learning for Symbolic Execution
• Channel-aware Decoupling Network for Multi-turn Dialogue Comprehension
• Neural Network Analytic Continuation for Monte Carlo: Improvement by Statistical Errors
• GlyphCRM: Bidirectional Encoder Representation for Chinese Character with its Glyph
• Residual U-Net for accurate and efficient prediction of hemodynamics in two-dimensional asymmetric stenosis
• RecSal : Deep Recursive Supervision for Visual Saliency Prediction
• Towards Socially Intelligent Agents with Mental State Transition and Human Utility
• Harvesting the Lyα forest with convolutional neural networks
• Bayesian RG Flow in Neural Network Field Theories
• Adaptive convolutional neural networks for k-space data interpolation in fast magnetic resonance imaging
• Quantum Neural Networks in Practice: A Comparative Study with Classical Models from Standard Data Sets to Industrial Images
• A Deep Learning Application for Psoriasis Detection
• Learning Incident Prediction Models Over Large Geographical Areas for Emergency Response Systems
• Content-Aware Preserving Image Generation
• Halo Properties from Observable Measures of Environment: I. Halo and Subhalo Masses
• Clusternets: A deep learning approach to probe clustering dark energy
• Context-aware Heterogeneous Graph Attention Network for User Behavior Prediction in Local Consumer Service Platform
• ICTSurF: Implicit Continuous-Time Survival Functions with Neural Networks
• Timing techniques with picosecond-order accuracy for novel gaseous detectors
• Particle Graph Autoencoders and Differentiable, Learned Energy Mover's Distance
• Dual-side Sparse Tensor Core
• Determinant- and Derivative-Free Quantum Monte Carlo Within the Stochastic Representation of Wavefunctions
• Learn distributed GAN with Temporary Discriminators
• Continuous Speech Recognition using EEG and Video
• Active Transfer Prototypical Network: An Efficient Labeling Algorithm for Time-Series Data
• Neural Networks for cosmological model selection and feature importance using Cosmic Microwave Background data
• Conspiracy to Commit: Information Pollution, Artificial Intelligence, and Real-World Hate Crime
• Learning quantum symmetries with interactive quantum-classical variational algorithms
• Systematic Study of Machine Learning Classification Algorithms of Zeolitic Imidazolate Framework Polymorphs
• Wizard of Search Engine: Access to Information Through Conversations with Search Engines
• Inter-channel Conv-TasNet for multichannel speech enhancement
• Memory Augmented Optimizers for Deep Learning
• Applying VertexShuffle Toward 360-Degree Video Super-Resolution on Focused-Icosahedral-Mesh
• From Hard to Soft: Understanding Deep Network Nonlinearities via Vector Quantization and Statistical Inference
• A Contactless Fingerprint Recognition System
• Inductive Matrix Completion Using Graph Autoencoder
• Transferring Voice Knowledge for Acoustic Event Detection: An Empirical Study
• Exploiting Inter-pixel Correlations in Unsupervised Domain Adaptation for Semantic Segmentation
• DeepDoseNet: A Deep Learning model for 3D Dose Prediction in Radiation Therapy
• AGNet: Weighing Black Holes with Machine Learning
• A Generative Model based Adversarial Security of Deep Learning and Linear Classifier Models
• A Differentially Private Multi-Output Deep Generative Networks Approach For Activity Diary Synthesis
• Few-Shot Website Fingerprinting Attack
• ALEX: Active Learning based Enhancement of a Model's Explainability
• Method for classifying a noisy Raman spectrum based on a wavelet transform and a deep neural network
• Post-Training Embedding Alignment for Decoupling Enrollment and Runtime Speaker Recognition Models
• IIITT@Dravidian-CodeMix-FIRE2021: Transliterate or translate? Sentiment analysis of code-mixed text in Dravidian languages
• Obstacle avoidance and path finding for mobile robot navigation
• EffCNet: An Efficient CondenseNet for Image Classification on NXP BlueBox
• Efficient Coding Approach Towards Non-Linear Spectro-Temporal Receptive Fields
• Will there be a construction? Predicting road constructions based on heterogeneous spatiotemporal data
• Research on Optimization Method of Multi-scale Fish Target Fast Detection Network
• Improving Multimodal fusion via Mutual Dependency Maximisation
• Sinogram Denoise Based on Generative Adversarial Networks
• Segmentation of Microscopy Data for finding Nuclei in Divergent Images
• Resource Constrained Neural Networks for 5G Direction-of-Arrival Estimation in Micro-controllers
• Understanding Ancient Coin Images
• A Novel Approach to Lifelong Learning: The Plastic Support Structure
• End-to-End Speech Emotion Recognition: Challenges of Real-Life Emergency Call Centers Data Recordings
• DNNs as Layers of Cooperating Classifiers
• Estimating Multiplicative Relations in Neural Networks
• Differential Deep Detection in Massive MIMO With One-Bit ADC
• SERE: Exploring Feature Self-relation for Self-supervised Transformer
• Ant Colony Inspired Machine Learning Algorithm for Identifying and Emulating Virtual Sensors
• Landmark-Aware and Part-based Ensemble Transfer Learning Network for Facial Expression Recognition from Static images
• Population Gradients improve performance across data-sets and architectures in object classification