Searching for Activation Functions

Abstract

The choice of activation functions in deep networks has a significant effect on the training dynamics and task performance. Currently, the most successful and widely-used activation function is the Rectified Linear Unit (ReLU). Although various hand-designed alternatives to ReLU have been proposed, none have managed to replace it due to inconsistent gains. In this work, we propose to leverage automatic search techniques to discover new activation functions. Using a combination of exhaustive and reinforcement learning-based search, we discover multiple novel activation functions. We verify the effectiveness of the searches by conducting an empirical evaluation with the best discovered activation function. Our experiments show that the best discovered activation function, $f(x)=x\cdot \text{sigmoid}(\beta x)$, which we name Swish, tends to work better than ReLU on deeper models across a number of challenging datasets. For example, simply replacing ReLUs with Swish units improves top-1 classification accuracy on ImageNet by 0.9\% for Mobile NASNet-A and 0.6\% for Inception-ResNet-v2. The simplicity of Swish and its similarity to ReLU make it easy for practitioners to replace ReLUs with Swish units in any neural network.

Updated version of "Swish: a Self-Gated Activation Function"

References in corpus (18)

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Deep Residual Learning for Image Recognition
• Neural Architecture Search with Reinforcement Learning
• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
• Empirical Evaluation of Rectified Activations in Convolutional Network
• Theoretical Models of Learning to Learn
• Language Modeling with Gated Convolutional Networks
• RL${}^2$: Fast Reinforcement Learning via Slow Reinforcement Learning
• Layer Normalization
• Learning to reinforcement learn
• Learning Activation Functions to Improve Deep Neural Networks
• Learnable pooling with Context Gating for video classification
• Neural Optimizer Search with Reinforcement Learning
• Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning
• On the Expressive Power of Deep Neural Networks
• Deep Interest Network for Click-Through Rate Prediction
• Practical Block-wise Neural Network Architecture Generation

Cited by in corpus (304)

• YOLOv4: Optimal Speed and Accuracy of Object Detection
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• AutoML: A Survey of the State-of-the-Art
• Surrogate Modeling for Fluid Flows Based on Physics-Constrained Deep Learning Without Simulation Data
• ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices
• A scoping review of transfer learning research on medical image analysis using ImageNet
• Deep UQ: Learning deep neural network surrogate models for high dimensional uncertainty quantification
• Hyper-Parameter Optimization: A Review of Algorithms and Applications
• Conformer: Convolution-augmented Transformer for Speech Recognition
• NVAE: A Deep Hierarchical Variational Autoencoder
• CSPNet: A New Backbone that can Enhance Learning Capability of CNN
• SpookyNet: Learning Force Fields with Electronic Degrees of Freedom and Nonlocal Effects
• MixConv: Mixed Depthwise Convolutional Kernels
• Reduced-order modeling of advection-dominated systems with recurrent neural networks and convolutional autoencoders
• The Principles of Deep Learning Theory
• Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation
• Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition
• Variational Encoding of Complex Dynamics
• Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models
• Design Space for Graph Neural Networks
• Simulator-free Solution of High-Dimensional Stochastic Elliptic Partial Differential Equations using Deep Neural Networks
• Physics-Informed Probabilistic Learning of Linear Embeddings of Non-linear Dynamics With Guaranteed Stability
• BS-Net: learning COVID-19 pneumonia severity on a large Chest X-Ray dataset
• Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examples
• Physics-guided deep learning framework for predictive modeling of the Reynolds stress anisotropy
• E(n) Equivariant Graph Neural Networks
• What Are Bayesian Neural Network Posteriors Really Like?
• Smooth Adversarial Training
• Machine Learning Holography for 3D Particle Field Imaging
• Meta-Learning Update Rules for Unsupervised Representation Learning
• Universal Planning Networks
• Deep Polynomial Neural Networks
• Logic Explained Networks
• H-NeRF: Neural Radiance Fields for Rendering and Temporal Reconstruction of Humans in Motion
• The Universal Approximation Property
• A Comparison of 1-D and 2-D Deep Convolutional Neural Networks in ECG Classification
• Constraint-Aware Neural Networks for Riemann Problems
• ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Context
• UNIT-DDPM: UNpaired Image Translation with Denoising Diffusion Probabilistic Models
• t-Soft Update of Target Network for Deep Reinforcement Learning
• Generalizing Convolutional Neural Networks for Equivariance to Lie Groups on Arbitrary Continuous Data
• EfficientPose: An efficient, accurate and scalable end-to-end 6D multi object pose estimation approach
• ECOVNet: An Ensemble of Deep Convolutional Neural Networks Based on EfficientNet to Detect COVID-19 From Chest X-rays
• Learning to Denoise Astronomical Images with U-nets
• A Derivative-Free Method for Solving Elliptic Partial Differential Equations with Deep Neural Networks
• Choose a Transformer: Fourier or Galerkin
• Universal Activation Function For Machine Learning
• Neural Networks Fail to Learn Periodic Functions and How to Fix It
• Neural Network Augmented Physics Models for Systems with Partially Unknown Dynamics: Application to Slider-Crank Mechanism
• Molecular Mechanics-Driven Graph Neural Network with Multiplex Graph for Molecular Structures
• Rotation Invariant Graph Neural Networks using Spin Convolutions
• Gait recognition via deep learning of the center-of-pressure trajectory
• One-Shot Hierarchical Imitation Learning of Compound Visuomotor Tasks
• MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learning
• Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradients
• dPOLY: Deep Learning of Polymer Phases and Phase Transition
• On the linearity of large non-linear models: when and why the tangent kernel is constant
• Activation Functions in Artificial Neural Networks: A Systematic Overview
• Explaining Explanations: Axiomatic Feature Interactions for Deep Networks
• Predictive Collective Variable Discovery with Deep Bayesian Models
• BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Models
• IDF++: Analyzing and Improving Integer Discrete Flows for Lossless Compression
• A fast multi-fidelity method with uncertainty quantification for complex data correlations: Application to vortex-induced vibrations of marine risers
• Toward Robust Autotuning of Noisy Quantum Dot Devices
• Learning the Stein Discrepancy for Training and Evaluating Energy-Based Models without Sampling
• Sign Language Recognition via Skeleton-Aware Multi-Model Ensemble
• NVIDIA SimNet^{TM}: an AI-accelerated multi-physics simulation framework
• Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Search
• Rice Classification Using Spatio-Spectral Deep Convolutional Neural Network
• FitVid: Overfitting in Pixel-Level Video Prediction
• Universal Approximation with Deep Narrow Networks
• Skeleton Aware Multi-modal Sign Language Recognition
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• Tied & Reduced RNN-T Decoder
• CSL-YOLO: A New Lightweight Object Detection System for Edge Computing
• Context-aware Dynamics Model for Generalization in Model-Based Reinforcement Learning
• A Practical Method for Constructing Equivariant Multilayer Perceptrons for Arbitrary Matrix Groups
• ForceNet: A Graph Neural Network for Large-Scale Quantum Calculations
• ProbAct: A Probabilistic Activation Function for Deep Neural Networks
• EENMF: An End-to-End Neural Matching Framework for E-Commerce Sponsored Search
• Dual-mode ASR: Unify and Improve Streaming ASR with Full-context Modeling
• Representation Matters: Offline Pretraining for Sequential Decision Making
• PLU: The Piecewise Linear Unit Activation Function
• SpineNet: Learning Scale-Permuted Backbone for Recognition and Localization
• Countdown Regression: Sharp and Calibrated Survival Predictions
• Neutron: An Implementation of the Transformer Translation Model and its Variants
• PaXNet: Dental Caries Detection in Panoramic X-ray using Ensemble Transfer Learning and Capsule Classifier
• GLU Variants Improve Transformer
• Gradient Descent can Learn Less Over-parameterized Two-layer Neural Networks on Classification Problems
• Generalised gravitational burst generation with Generative Adversarial Networks
• Perpetual Motion: Generating Unbounded Human Motion
• Supply-Power-Constrained Cable Capacity Maximization Using Multi-Layer Neural Networks
• HARK Side of Deep Learning -- From Grad Student Descent to Automated Machine Learning
• STAR: Sparse Transformer-based Action Recognition
• Stein variational gradient descent with local approximations
• LSQ+: Improving low-bit quantization through learnable offsets and better initialization
• Rethinking Co-design of Neural Architectures and Hardware Accelerators
• MoViNets: Mobile Video Networks for Efficient Video Recognition
• Soft-Root-Sign Activation Function
• Optimal Rates for Averaged Stochastic Gradient Descent under Neural Tangent Kernel Regime
• Meta-learning curiosity algorithms
• Angle-based Search Space Shrinking for Neural Architecture Search
• Physics-Constrained Bayesian Neural Network for Fluid Flow Reconstruction with Sparse and Noisy Data
• Funnel Activation for Visual Recognition
• W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training
• IDRLnet: A Physics-Informed Neural Network Library
• AReLU: Attention-based Rectified Linear Unit
• Using a thousand optimization tasks to learn hyperparameter search strategies
• Rethinking Channel Dimensions for Efficient Model Design
• Rotation-Equivariant Deep Learning for Diffusion MRI
• Expressive Priors in Bayesian Neural Networks: Kernel Combinations and Periodic Functions
• CKConv: Continuous Kernel Convolution For Sequential Data
• A Methodology for Automatic Selection of Activation Functions to Design Hybrid Deep Neural Networks
• An unsupervised deep learning approach in solving partial integro-differential equations
• Locally-symplectic neural networks for learning volume-preserving dynamics
• Neural-Guided Symbolic Regression with Asymptotic Constraints
• A White Paper on Neural Network Quantization
• SUNLayer: Stable denoising with generative networks
• Geometric and Physical Quantities Improve E(3) Equivariant Message Passing
• BETANAS: BalancEd TrAining and selective drop for Neural Architecture Search
• Finding quadruply imaged quasars with machine learning. I. Methods
• Primer: Searching for Efficient Transformers for Language Modeling
• How to Learn a Useful Critic? Model-based Action-Gradient-Estimator Policy Optimization
• DVOLVER: Efficient Pareto-Optimal Neural Network Architecture Search
• Parallel Tacotron 2: A Non-Autoregressive Neural TTS Model with Differentiable Duration Modeling
• The effect of Target Normalization and Momentum on Dying ReLU
• Can weight sharing outperform random architecture search? An investigation with TuNAS
• GradInit: Learning to Initialize Neural Networks for Stable and Efficient Training
• Training Larger Networks for Deep Reinforcement Learning
• Neural Empirical Bayes
• A3D: Adaptive 3D Networks for Video Action Recognition
• An adaptive surrogate modeling based on deep neural networks for large-scale Bayesian inverse problems
• Activate or Not: Learning Customized Activation
• Adaptive and Multiple Time-scale Eligibility Traces for Online Deep Reinforcement Learning
• Revealing and Protecting Labels in Distributed Training
• A New Backpropagation Algorithm without Gradient Descent
• A Contrastive Learning Approach for Training Variational Autoencoder Priors
• On the detectability of ultra-compact binary pulsar systems
• Knowledge-based Radiation Treatment Planning: A Data-driven Method Survey
• Supermasks in Superposition
• Classification of Diabetic Retinopathy via Fundus Photography: Utilization of Deep Learning Approaches to Speed up Disease Detection
• Diff-TTS: A Denoising Diffusion Model for Text-to-Speech
• Complexity Measures for Neural Networks with General Activation Functions Using Path-based Norms
• Oops I Took A Gradient: Scalable Sampling for Discrete Distributions
• Selection dynamics for deep neural networks
• Compositional Visual Generation and Inference with Energy Based Models
• A Note on Deepfake Detection with Low-Resources
• Regularized Binary Network Training
• S3NAS: Fast NPU-aware Neural Architecture Search Methodology
• Weighted Sigmoid Gate Unit for an Activation Function of Deep Neural Network
• Symmetry & critical points for a model shallow neural network
• Trust Region Based Adversarial Attack on Neural Networks
• Activation function impact on Sparse Neural Networks
• Compiler-Level Matrix Multiplication Optimization for Deep Learning
• Learning Barrier Certificates: Towards Safe Reinforcement Learning with Zero Training-time Violations
• Learning to Generate Synthetic Data via Compositing
• Is it time to swish? Comparing activation functions in solving the Helmholtz equation using physics-informed neural networks
• Likelihood Training of Schrödinger Bridge using Forward-Backward SDEs Theory
• HourNAS: Extremely Fast Neural Architecture Search Through an Hourglass Lens
• An Integrated Deep Learning and Dynamic Programming Method for Predicting Tumor Suppressor Genes, Oncogenes, and Fusion from PDB Structures
• Interpreting Neural Networks Using Flip Points
• Regularizing Trajectory Optimization with Denoising Autoencoders
• Richer priors for infinitely wide multi-layer perceptrons
• Evolving Search Space for Neural Architecture Search
• Neural Spectrahedra and Semidefinite Lifts: Global Convex Optimization of Polynomial Activation Neural Networks in Fully Polynomial-Time
• Zero-shot Imitation Learning from Demonstrations for Legged Robot Visual Navigation
• Deep neural network for solving differential equations motivated by Legendre-Galerkin approximation
• Machine Learning Surrogates for Predicting Response of an Aero-Structural-Sloshing System
• Geometry-Aware Gradient Algorithms for Neural Architecture Search
• Learning Compact Neural Networks Using Ordinary Differential Equations as Activation Functions
• MUXConv: Information Multiplexing in Convolutional Neural Networks
• A Medical Pre-Diagnosis System for Histopathological Image of Breast Cancer
• TRAIL: Near-Optimal Imitation Learning with Suboptimal Data
• Anomaly scores for generative models
• Effectiveness of Scaled Exponentially-Regularized Linear Units (SERLUs)
• Neural Spatio-Temporal Point Processes
• VA-RED${}^2$: Video Adaptive Redundancy Reduction
• Quasi-Monte Carlo sampling for machine-learning partial differential equations
• A Picture May Be Worth a Hundred Words for Visual Question Answering
• Bag of Tricks for Neural Architecture Search
• SeesawFaceNets: sparse and robust face verification model for mobile platform
• Natural-Logarithm-Rectified Activation Function in Convolutional Neural Networks
• A High-Performance Adaptive Quantization Approach for Edge CNN Applications
• Implicit Generative Modeling for Efficient Exploration
• Symmetrical Gaussian Error Linear Units (SGELUs)
• Alternating Direction Method of Multipliers for Quantization
• Searching for Fast Model Families on Datacenter Accelerators
• MHVAE: a Human-Inspired Deep Hierarchical Generative Model for Multimodal Representation Learning
• TanhSoft -- a family of activation functions combining Tanh and Softplus
• Human-guided Robot Behavior Learning: A GAN-assisted Preference-based Reinforcement Learning Approach
• On the Universality of the Double Descent Peak in Ridgeless Regression
• Group Equivariant Stand-Alone Self-Attention For Vision
• Input Hessian Regularization of Neural Networks
• Meta-Model-Based Meta-Policy Optimization
• Sisyphus: A Cautionary Tale of Using Low-Degree Polynomial Activations in Privacy-Preserving Deep Learning
• Deep Learning for direct Dark Matter search with nuclear emulsions
• K-TanH: Efficient TanH For Deep Learning
• Masking Neural Networks Using Reachability Graphs to Predict Process Events
• Density Ratio Estimation via Infinitesimal Classification
• Piecewise Linear Units Improve Deep Neural Networks
• Synthesizing Irreproducibility in Deep Networks
• Activated Gradients for Deep Neural Networks
• HR-NAS: Searching Efficient High-Resolution Neural Architectures with Lightweight Transformers
• Highly Accurate CNN Inference Using Approximate Activation Functions over Homomorphic Encryption
• DiffSVC: A Diffusion Probabilistic Model for Singing Voice Conversion
• MUFASA: Multimodal Fusion Architecture Search for Electronic Health Records
• Nonconvex sparse regularization for deep neural networks and its optimality
• Physics-Informed Neural Networks for Solving Multiscale Mode-Resolved Phonon Boltzmann Transport Equation
• LiDAR ICPS-net: Indoor Camera Positioning based-on Generative Adversarial Network for RGB to Point-Cloud Translation
• A Strong Baseline for Crowd Counting and Unsupervised People Localization
• RandomForestMLP: An Ensemble-Based Multi-Layer Perceptron Against Curse of Dimensionality
• Exploiting Activation based Gradient Output Sparsity to Accelerate Backpropagation in CNNs
• FixNorm: Dissecting Weight Decay for Training Deep Neural Networks
• GroupBERT: Enhanced Transformer Architecture with Efficient Grouped Structures
• DelightfulTTS: The Microsoft Speech Synthesis System for Blizzard Challenge 2021
• SMU: smooth activation function for deep networks using smoothing maximum technique
• Non-autoregressive sequence-to-sequence voice conversion
• GeoT: A Geometry-aware Transformer for Reliable Molecular Property Prediction and Chemically Interpretable Representation Learning
• The use of Generative Adversarial Networks to characterise new physics in multi-lepton final states at the LHC
• Improving correlation method with convolutional neural networks
• Smoothness Analysis of Adversarial Training
• Efficient Scale-Permuted Backbone with Learned Resource Distribution
• Learning of Discrete Graphical Models with Neural Networks
• Attention-Based Face AntiSpoofing of RGB Images, using a Minimal End-2-End Neural Network
• Infinitely deep neural networks as diffusion processes
• Local Temporal Bilinear Pooling for Fine-grained Action Parsing
• Efficient Design of Neural Networks with Random Weights
• Regularized Flexible Activation Function Combinations for Deep Neural Networks
• ARM: A Confidence-Based Adversarial Reweighting Module for Coarse Semantic Segmentation
• Machine Learning Approach for Transforming Scattering Parameters to Complex Permittivity
• NeuralScale: Efficient Scaling of Neurons for Resource-Constrained Deep Neural Networks
• Multi-Activation Hidden Units for Neural Networks with Random Weights
• Supply-Power-Constrained Cable Capacity Maximization Using Deep Neural Networks
• Learning and Inference in Imaginary Noise Models
• Spending Your Winning Lottery Better After Drawing It
• Model-based Multi-agent Policy Optimization with Adaptive Opponent-wise Rollouts
• Deep limits and cut-off phenomena for neural networks
• Convergent Graph Solvers
• Molecular modeling with machine-learned universal potential functions
• Homological Time Series Analysis of Sensor Signals from Power Plants
• Continual Backprop: Stochastic Gradient Descent with Persistent Randomness
• AutoShrink: A Topology-aware NAS for Discovering Efficient Neural Architecture
• Predicting Effective Diffusivity of Porous Media from Images by Deep Learning
• A New Multifractal-based Deep Learning Model for Text Mining
• Parametric Rectified Power Sigmoid Units: Learning Nonlinear Neural Transfer Analytical Forms
• SyMetric: Measuring the Quality of Learnt Hamiltonian Dynamics Inferred from Vision
• Borrowing from Similar Code: A Deep Learning NLP-Based Approach for Log Statement Automation
• Learning Large-Time-Step Molecular Dynamics with Graph Neural Networks
• Raw Waveform Encoder with Multi-Scale Globally Attentive Locally Recurrent Networks for End-to-End Speech Recognition
• Orthogonal-Padé Activation Functions: Trainable Activation functions for smooth and faster convergence in deep networks
• Wasserstein Introspective Neural Networks
• Measure, Manifold, Learning, and Optimization: A Theory Of Neural Networks
• Multikernel activation functions: formulation and a case study
• Instance Segmentation Challenge Track Technical Report, VIPriors Workshop at ICCV 2021: Task-Specific Copy-Paste Data Augmentation Method for Instance Segmentation
• Sample-based Federated Learning via Mini-batch SSCA
• MOBA-Slice: A Time Slice Based Evaluation Framework of Relative Advantage between Teams in MOBA Games
• Exploring the potential of transfer learning for metamodels of heterogeneous material deformation
• Deep Pepper: Expert Iteration based Chess agent in the Reinforcement Learning Setting
• SIPA: A Simple Framework for Efficient Networks
• Fetal MRI by robust deep generative prior reconstruction and diffeomorphic registration: application to gestational age prediction
• Light Multi-segment Activation for Model Compression
• Automatic design of novel potential 3CL${}^{\text{pro}}$ and PL${}^{\text{pro}}$ inhibitors
• ISyNet: Convolutional Neural Networks design for AI accelerator
• Parametric Variational Linear Units (PVLUs) in Deep Convolutional Networks
• An acceleration strategy for randomize-then-optimize sampling via deep neural networks
• Pricing spread option with liquidity adjustments
• Unbounded Output Networks for Classification
• Analysis of key flavors of event-driven predictive maintenance using logs of phenomena described by Weibull distributions
• FSD: Feature Skyscraper Detector for Stem End and Blossom End of Navel Orange
• Particle Dynamics for Learning EBMs
• Automatically Designing CNN Architectures for Medical Image Segmentation
• Latent Transformations for Object View Points Synthesis
• Neural Network Structure Design based on N-Gauss Activation Function
• Unnormalized Variational Bayes
• Cross-individual Recognition of Emotions by a Dynamic Entropy based on Pattern Learning with EEG features
• A Wasserstein Minimum Velocity Approach to Learning Unnormalized Models
• JNR: Joint-based Neural Rig Representation for Compact 3D Face Modeling
• EIS -- a family of activation functions combining Exponential, ISRU, and Softplus
• RoSearch: Search for Robust Student Architectures When Distilling Pre-trained Language Models
• Goldilocks Neural Networks
• Speaker Representation Learning using Global Context Guided Channel and Time-Frequency Transformations
• When deep learning models on GPU can be accelerated by taking advantage of unstructured sparsity
• Avoiding Kernel Fixed Points: Computing with ELU and GELU Infinite Networks
• Highway Transformer: Self-Gating Enhanced Self-Attentive Networks
• Constraining Logits by Bounded Function for Adversarial Robustness
• RLCache: Automated Cache Management Using Reinforcement Learning
• Research on Optimization Method of Multi-scale Fish Target Fast Detection Network
• Searching by Generating: Flexible and Efficient One-Shot NAS with Architecture Generator
• Deep Learning Order Parameter for Polymer Phase Transition
• Generation and Simulation of Yeast Microscopy Imagery with Deep Learning
• The SWAG Algorithm; a Mathematical Approach that Outperforms Traditional Deep Learning. Theory and Implementation
• FA-GAN: Fused Attentive Generative Adversarial Networks for MRI Image Super-Resolution
• Using Topological Framework for the Design of Activation Function and Model Pruning in Deep Neural Networks
• Learning of feature points without additional supervision improves reinforcement learning from images
• Application of Facial Recognition using Convolutional Neural Networks for Entry Access Control
• Meta-Learning with Variational Bayes
• Random Bias Initialization Improves Quantized Training
• Spatial light interference microscopy (SLIM): principle and applications to biomedicine
• Deep Learning for Estimating Synaptic Health of Primary Neuronal Cell Culture
• HPTQ: Hardware-Friendly Post Training Quantization
• Rediscovering Deep Neural Networks Through Finite-State Distributions
• Introducing the DOME Activation Functions
• Deep Learning for CSI Feedback Based on Superimposed Coding
• A survey on deep learning approaches for breast cancer diagnosis