Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)

Abstract

We introduce the "exponential linear unit" (ELU) which speeds up learning in deep neural networks and leads to higher classification accuracies. Like rectified linear units (ReLUs), leaky ReLUs (LReLUs) and parametrized ReLUs (PReLUs), ELUs alleviate the vanishing gradient problem via the identity for positive values. However, ELUs have improved learning characteristics compared to the units with other activation functions. In contrast to ReLUs, ELUs have negative values which allows them to push mean unit activations closer to zero like batch normalization but with lower computational complexity. Mean shifts toward zero speed up learning by bringing the normal gradient closer to the unit natural gradient because of a reduced bias shift effect. While LReLUs and PReLUs have negative values, too, they do not ensure a noise-robust deactivation state. ELUs saturate to a negative value with smaller inputs and thereby decrease the forward propagated variation and information. Therefore, ELUs code the degree of presence of particular phenomena in the input, while they do not quantitatively model the degree of their absence. In experiments, ELUs lead not only to faster learning, but also to significantly better generalization performance than ReLUs and LReLUs on networks with more than 5 layers. On CIFAR-100 ELUs networks significantly outperform ReLU networks with batch normalization while batch normalization does not improve ELU networks. ELU networks are among the top 10 reported CIFAR-10 results and yield the best published result on CIFAR-100, without resorting to multi-view evaluation or model averaging. On ImageNet, ELU networks considerably speed up learning compared to a ReLU network with the same architecture, obtaining less than 10% classification error for a single crop, single model network.

Published as a conference paper at ICLR 2016

References in corpus (6)

• Striving for Simplicity: The All Convolutional Net
• Fractional Max-Pooling
• Toxicity Prediction using Deep Learning
• Learning Feature Hierarchies with Centered Deep Boltzmann Machines
• Natural Neural Networks
• Metric-Free Natural Gradient for Joint-Training of Boltzmann Machines

Cited by in corpus (585)

• Deep learning with convolutional neural networks for EEG decoding and visualization
• Wide Residual Networks
• Activation Functions: Comparison of trends in Practice and Research for Deep Learning
• Searching for Activation Functions
• BEGAN: Boundary Equilibrium Generative Adversarial Networks
• f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization
• A comprehensive review on convolutional neural network in machine fault diagnosis
• DeepInf: Social Influence Prediction with Deep Learning
• Group Equivariant Convolutional Networks
• WeatherBench: A benchmark dataset for data-driven weather forecasting
• Deep UQ: Learning deep neural network surrogate models for high dimensional uncertainty quantification
• From Big to Small: Multi-Scale Local Planar Guidance for Monocular Depth Estimation
• Discovering physical concepts with neural networks
• SchNet: A continuous-filter convolutional neural network for modeling quantum interactions
• End-to-End Unsupervised Deformable Image Registration with a Convolutional Neural Network
• Residual Networks of Residual Networks: Multilevel Residual Networks
• Artificial neural networks for neuroscientists: A primer
• NVAE: A Deep Hierarchical Variational Autoencoder
• Xception: Deep Learning with Depthwise Separable Convolutions
• A Light CNN for detecting COVID-19 from CT scans of the chest
• Certifying Some Distributional Robustness with Principled Adversarial Training
• Systematic evaluation of CNN advances on the ImageNet
• Contrastive Learning of Subject-Invariant EEG Representations for Cross-Subject Emotion Recognition
• Causal Effect Inference with Deep Latent-Variable Models
• Extremely Large Minibatch SGD: Training ResNet-50 on ImageNet in 15 Minutes
• CANet: An Unsupervised Intrusion Detection System for High Dimensional CAN Bus Data
• A Survey of Convolutional Neural Networks: Analysis, Applications, and Prospects
• Learning by Playing - Solving Sparse Reward Tasks from Scratch
• Multi-Object Representation Learning with Iterative Variational Inference
• How Does Batch Normalization Help Optimization?
• Superhuman Accuracy on the SNEMI3D Connectomics Challenge
• Deep learning analysis of the myocardium in coronary CT angiography for identification of patients with functionally significant coronary artery stenosis
• Generative Image Inpainting with Contextual Attention
• Unsupervised Scale-consistent Depth Learning from Video
• ECG arrhythmia classification using a 2-D convolutional neural network
• Extending the Bump Hunt with Machine Learning
• EikoNet: Solving the Eikonal equation with Deep Neural Networks
• EddyNet: A Deep Neural Network For Pixel-Wise Classification of Oceanic Eddies
• Learning Constitutive Relations from Indirect Observations Using Deep Neural Networks
• Improving Variational Inference with Inverse Autoregressive Flow
• Invertible Residual Networks
• The Strong Gravitational Lens Finding Challenge
• StegNet: Mega Image Steganography Capacity with Deep Convolutional Network
• A Comprehensive Overview and Comparative Analysis on Deep Learning Models: CNN, RNN, LSTM, GRU
• Satellite Imagery Feature Detection using Deep Convolutional Neural Network: A Kaggle Competition
• Deep Learning How to Fit an Intravoxel Incoherent Motion Model to Diffusion-Weighted MRI
• A Compact and Interpretable Convolutional Neural Network for Cross-Subject Driver Drowsiness Detection from Single-Channel EEG
• UAV and Machine Learning Based Refinement of a Satellite-Driven Vegetation Index for Precision Agriculture
• Hierarchical Scene Coordinate Classification and Regression for Visual Localization
• Improved Techniques for Training Score-Based Generative Models
• Few-Shot Learning with Metric-Agnostic Conditional Embeddings
• Dream to Control: Learning Behaviors by Latent Imagination
• Compression-aware Training of Deep Networks
• Direct Automatic Coronary Calcium Scoring in Cardiac and Chest CT
• Transfer learning for music classification and regression tasks
• Deep Chain HDRI: Reconstructing a High Dynamic Range Image from a Single Low Dynamic Range Image
• Explaining by Removing: A Unified Framework for Model Explanation
• Recent Advances in Deep Learning Techniques for Face Recognition
• Optimize TSK Fuzzy Systems for Classification Problems: Mini-Batch Gradient Descent with Uniform Regularization and Batch Normalization
• Comparison of non-linear activation functions for deep neural networks on MNIST classification task
• PEPSI++: Fast and Lightweight Network for Image Inpainting
• Deep Residual Networks with Exponential Linear Unit
• TeaNet: universal neural network interatomic potential inspired by iterative electronic relaxations
• Machine Learning the Physical Non-Local Exchange-Correlation Functional of Density-Functional Theory
• B-CNN: Branch Convolutional Neural Network for Hierarchical Classification
• Comparison of Deep Learning and the Classical Machine Learning Algorithm for the Malware Detection
• Semi-supervised Learning with GANs: Manifold Invariance with Improved Inference
• Review: Deep Learning in Electron Microscopy
• Entropy-SGD: Biasing Gradient Descent Into Wide Valleys
• Continuously Differentiable Exponential Linear Units
• The Conditional Entropy Bottleneck
• Deep 6-DOF Tracking
• Robust deep learning-based semantic organ segmentation in hyperspectral images
• Dissecting Neural ODEs
• Machine and Deep Learning for IoT Security and Privacy: Applications, Challenges, and Future Directions
• Improving the Generalization of Adversarial Training with Domain Adaptation
• Efficient Hyperparameter Optimization in Deep Learning Using a Variable Length Genetic Algorithm
• Metadynamics for Training Neural Network Model Chemistries: a Competitive Assessment
• A Deep Learning-based Reconstruction of Cosmic Ray-induced Air Showers
• Distance-based Confidence Score for Neural Network Classifiers
• Inference Suboptimality in Variational Autoencoders
• Predicting novel superconducting hydrides using machine learning approaches
• Preserving Semantic Relations for Zero-Shot Learning
• ChemGrapher: Optical Graph Recognition of Chemical Compounds by Deep Learning
• DiracNets: Training Very Deep Neural Networks Without Skip-Connections
• An autoencoder-based reduced-order model for eigenvalue problems with application to neutron diffusion
• Constraint-Aware Neural Networks for Riemann Problems
• Distance-based Self-Attention Network for Natural Language Inference
• Modeling Others using Oneself in Multi-Agent Reinforcement Learning
• Training Deep AutoEncoders for Collaborative Filtering
• On Deep Learning Techniques to Boost Monocular Depth Estimation for Autonomous Navigation
• Acoustic Anomaly Detection for Machine Sounds based on Image Transfer Learning
• Convolutional Recurrent Neural Networks for Music Classification
• Universal Activation Function For Machine Learning
• Neural Network Models for the Anisotropic Reynolds Stress Tensor in Turbulent Channel Flow
• Analysis and Optimization of Convolutional Neural Network Architectures
• On Loss Functions for Deep Neural Networks in Classification
• Neural Networks Fail to Learn Periodic Functions and How to Fix It
• Average Biased ReLU Based CNN Descriptor for Improved Face Retrieval
• Deep Hyperspherical Learning
• AdaTask: A Task-aware Adaptive Learning Rate Approach to Multi-task Learning
• A guide for deploying Deep Learning in LHC searches: How to achieve optimality and account for uncertainty
• Deciding How to Decide: Dynamic Routing in Artificial Neural Networks
• Listening for Sirens: Locating and Classifying Acoustic Alarms in City Scenes
• Lets keep it simple, Using simple architectures to outperform deeper and more complex architectures
• Learning by Association - A versatile semi-supervised training method for neural networks
• Automatic Seismic Salt Interpretation with Deep Convolutional Neural Networks
• Are we really making much progress? Revisiting, benchmarking, and refining heterogeneous graph neural networks
• Neural Network Reconstruction of Late-Time Cosmology and Null Tests
• Introduction to Camera Pose Estimation with Deep Learning
• Decoding EEG Brain Activity for Multi-Modal Natural Language Processing
• Forecasting Megaelectron-Volt Electrons inside Earth's Outer Radiation Belt: PreMevE 2.0 Based on Supervised Machine Learning Algorithms
• Automated detection of vulnerable plaque in intravascular ultrasound images
• A Survey on Deep Learning Methods for Robot Vision
• The importance of better models in stochastic optimization
• AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence
• Learning Gradient Descent: Better Generalization and Longer Horizons
• Variational auto-encoding of protein sequences
• Comparing Weak- and Unsupervised Methods for Resonant Anomaly Detection
• Fully Convolutional Mesh Autoencoder using Efficient Spatially Varying Kernels
• Efficient Processing of Deep Neural Networks: A Tutorial and Survey
• A Learning Strategy for Contrast-agnostic MRI Segmentation
• Deep Architectures for Automated Seizure Detection in Scalp EEGs
• Deep Learning Convolutional Networks for Multiphoton Microscopy Vasculature Segmentation
• Revise Saturated Activation Functions
• A Deep Learning Algorithm for High-Dimensional Exploratory Item Factor Analysis
• Crowding Reveals Fundamental Differences in Local vs. Global Processing in Humans and Machines
• Improving Deep Learning by Inverse Square Root Linear Units (ISRLUs)
• Separating the EoR Signal with a Convolutional Denoising Autoencoder: A Deep-learning-based Method
• Value Prediction Network
• A simple and efficient architecture for trainable activation functions
• Scanning Probe State Recognition With Multi-Class Neural Network Ensembles
• Neural 3D Morphable Models: Spiral Convolutional Networks for 3D Shape Representation Learning and Generation
• Acoustic Scene Classification with Squeeze-Excitation Residual Networks
• Neural Probabilistic Model for Non-projective MST Parsing
• AC-VRNN: Attentive Conditional-VRNN for Multi-Future Trajectory Prediction
• DVAE++: Discrete Variational Autoencoders with Overlapping Transformations
• Multi-Residual Networks: Improving the Speed and Accuracy of Residual Networks
• Flatten-T Swish: a thresholded ReLU-Swish-like activation function for deep learning
• Speaker Diarization using Deep Recurrent Convolutional Neural Networks for Speaker Embeddings
• Understanding GANs: the LQG Setting
• On the Loss Landscape of Adversarial Training: Identifying Challenges and How to Overcome Them
• Learning Pair Potentials using Differentiable Simulations
• Graph-Based Intercategory and Intermodality Network for Multilabel Classification and Melanoma Diagnosis of Skin Lesions in Dermoscopy and Clinical Images
• Identifying and Categorizing Anomalies in Retinal Imaging Data
• Gaussian Material Synthesis
• Deep Convolutional Neural Networks with Merge-and-Run Mappings
• Temporal Generative Adversarial Nets with Singular Value Clipping
• D3VO: Deep Depth, Deep Pose and Deep Uncertainty for Monocular Visual Odometry
• Oriented Response Networks
• Decoupling Dynamics and Reward for Transfer Learning
• The NN-Stacking: Feature weighted linear stacking through neural networks
• Robust Medical Instrument Segmentation Challenge 2019
• On the design space between molecular mechanics and machine learning force fields
• Unsupervised Monocular Depth Estimation with Left-Right Consistency
• Doubly Convolutional Neural Networks
• MRI Super-Resolution with GAN and 3D Multi-Level DenseNet: Smaller, Faster, and Better
• Solving Bayesian Inverse Problems via Variational Autoencoders
• CASI: A Convolutional Neural Network Approach for Shell Identification
• When is invariance useful in an Out-of-Distribution Generalization problem ?
• Contextual Residual Aggregation for Ultra High-Resolution Image Inpainting
• C-MIL: Continuation Multiple Instance Learning for Weakly Supervised Object Detection
• Enforcing constraints for interpolation and extrapolation in Generative Adversarial Networks
• Semantic and Relational Spaces in Science of Science: Deep Learning Models for Article Vectorisation
• The DNNLikelihood: enhancing likelihood distribution with Deep Learning
• What Do Neural Networks Learn When Trained With Random Labels?
• Propositionalization and Embeddings: Two Sides of the Same Coin
• Lipschitz constant estimation of Neural Networks via sparse polynomial optimization
• Likelihood-free MCMC with Amortized Approximate Ratio Estimators
• Throughput Optimizations for FPGA-based Deep Neural Network Inference
• Convolutions on Spherical Images
• Explainable machine learning of the underlying physics of high-energy particle collisions
• GlobeNet: Convolutional Neural Networks for Typhoon Eye Tracking from Remote Sensing Imagery
• Generative machine learning methods for multivariate ensemble post-processing
• Deep learning within a priori temporal feature spaces for large-scale dynamic MR image reconstruction: Application to 5-D cardiac MR Multitasking
• GraphCL: Contrastive Self-Supervised Learning of Graph Representations
• CNN-based Landmark Detection in Cardiac CTA Scans
• New Benchmarks for Learning on Non-Homophilous Graphs
• Improving the Neural GPU Architecture for Algorithm Learning
• A data-driven approach for the closure of RANS models by the divergence of the Reynolds Stress Tensor
• SLAPS: Self-Supervision Improves Structure Learning for Graph Neural Networks
• Machine Learning Universal Bosonic Functionals
• ProbAct: A Probabilistic Activation Function for Deep Neural Networks
• Asteroid Flyby Cycler Trajectory Design Using Deep Neural Networks
• Fast LIDAR-based Road Detection Using Fully Convolutional Neural Networks
• Hierarchically modelling Kepler dwarfs and subgiants to improve inference of stellar properties with asteroseismology
• A Convolutional Neural Network based Live Object Recognition System as Blind Aid
• EENMF: An End-to-End Neural Matching Framework for E-Commerce Sponsored Search
• Sample Mixed-Based Data Augmentation for Domestic Audio Tagging
• Video action detection by learning graph-based spatio-temporal interactions
• Dr.VAE: Drug Response Variational Autoencoder
• On Catastrophic Forgetting and Mode Collapse in Generative Adversarial Networks
• Acting Thoughts: Towards a Mobile Robotic Service Assistant for Users with Limited Communication Skills
• Mastering Atari with Discrete World Models
• Quantitative Phase Imaging and Artificial Intelligence: A Review
• Navigating by Touch: Haptic Monte Carlo Localization via Geometric Sensing and Terrain Classification
• A multi-branch convolutional neural network for detecting double JPEG compression
• Exact constraints and appropriate norms in machine learned exchange-correlation functionals
• Graph-based Solutions with Residuals for Intrusion Detection: the Modified E-GraphSAGE and E-ResGAT Algorithms
• Fine-Grained Visual Classification via Progressive Multi-Granularity Training of Jigsaw Patches
• Learning magnetization dynamics
• A General Pipeline for 3D Detection of Vehicles
• Now Playing: Continuous low-power music recognition
• Task-Relevant Adversarial Imitation Learning
• Learning Chinese Word Representations From Glyphs Of Characters
• Generating Multiple Hypotheses for 3D Human Pose Estimation with Mixture Density Network
• Quality-Net: An End-to-End Non-intrusive Speech Quality Assessment Model based on BLSTM
• Network of Experts for Large-Scale Image Categorization
• Self-Supervised Variational Auto-Encoders
• First-order Convergence Theory for Weakly-Convex-Weakly-Concave Min-max Problems
• Machine Learning for Observables: Reactant to Product State Distributions for Atom-Diatom Collisions
• Self-supervised EEG Representation Learning for Automatic Sleep Staging
• Residual or Gate? Towards Deeper Graph Neural Networks for Inductive Graph Representation Learning
• Multi-hop Convolutions on Weighted Graphs
• Fast and Provable ADMM for Learning with Generative Priors
• Gaussian Attention Model and Its Application to Knowledge Base Embedding and Question Answering
• Speech Recognition: Keyword Spotting Through Image Recognition
• HashTran-DNN: A Framework for Enhancing Robustness of Deep Neural Networks against Adversarial Malware Samples
• Deep Image Harmonization
• Chinese Typography Transfer
• Neumann Networks for Inverse Problems in Imaging
• Learnable Explicit Density for Continuous Latent Space and Variational Inference
• Benchmarking Regression Methods: A comparison with CGAN
• Completing density functional theory by machine-learning hidden messages from molecules
• ComDefend: An Efficient Image Compression Model to Defend Adversarial Examples
• Inverse Modeling of Viscoelasticity Materials using Physics Constrained Learning
• Transfer Learning of Artist Group Factors to Musical Genre Classification
• Learning Audio - Sheet Music Correspondences for Score Identification and Offline Alignment
• Deep Recurrent Convolutional Neural Network: Improving Performance For Speech Recognition
• VRSTC: Occlusion-Free Video Person Re-Identification
• Soft-Root-Sign Activation Function
• Global Planar Convolutions for improved context aggregation in Brain Tumor Segmentation
• Keep the Gradients Flowing: Using Gradient Flow to Study Sparse Network Optimization
• Adjusting for Dropout Variance in Batch Normalization and Weight Initialization
• Deep learning for photoacoustic imaging: a survey
• A Novel Framework Integrating AI Model and Enzymological Experiments Promotes Identification of SARS-CoV-2 3CL Protease Inhibitors and Activity-based Probe
• The Riemannian Geometry of Deep Generative Models
• Accelerating PDE-constrained Inverse Solutions with Deep Learning and Reduced Order Models
• Empirical analysis of non-linear activation functions for Deep Neural Networks in classification tasks
• Funnel Activation for Visual Recognition
• Towards "AlphaChem": Chemical Synthesis Planning with Tree Search and Deep Neural Network Policies
• FCOSR: A Simple Anchor-free Rotated Detector for Aerial Object Detection
• Randomized Entity-wise Factorization for Multi-Agent Reinforcement Learning
• Data Dropout: Optimizing Training Data for Convolutional Neural Networks
• Learning to Synthesize a 4D RGBD Light Field from a Single Image
• Exploring the Design Space of Deep Convolutional Neural Networks at Large Scale
• Riemannian approach to batch normalization
• A New Distance Measure for Non-Identical Data with Application to Image Classification
• AReLU: Attention-based Rectified Linear Unit
• Deep Outdoor Illumination Estimation
• E-swish: Adjusting Activations to Different Network Depths
• A Methodology for Automatic Selection of Activation Functions to Design Hybrid Deep Neural Networks
• On the energy landscape of deep networks
• Frank-Wolfe Network: An Interpretable Deep Structure for Non-Sparse Coding
• Harnessing the Power of Infinitely Wide Deep Nets on Small-data Tasks
• Single Image Super-resolution via a Lightweight Residual Convolutional Neural Network
• Federated Learning for Malware Detection in IoT Devices
• An Error Detection and Correction Framework for Connectomics
• Range Conditioned Dilated Convolutions for Scale Invariant 3D Object Detection
• Speaker attribution with voice profiles by graph-based semi-supervised learning
• Stagewise Training Accelerates Convergence of Testing Error Over SGD
• Learning to Listen, Read, and Follow: Score Following as a Reinforcement Learning Game
• Content-Based Citation Recommendation
• SUNLayer: Stable denoising with generative networks
• Joint Learning for Pulmonary Nodule Segmentation, Attributes and Malignancy Prediction
• A* CCG Parsing with a Supertag and Dependency Factored Model
• SPAGAN: Shortest Path Graph Attention Network
• Pairing Character Classes in a Deathmatch Shooter Game via a Deep-Learning Surrogate Model
• Cross-domain Recommendation via Deep Domain Adaptation
• Weakly- and Self-Supervised Learning for Content-Aware Deep Image Retargeting
• An Explainable Deep Learning-based Prognostic Model for Rotating Machinery
• Convolutional Residual Memory Networks
• Recent Advances in the Applications of Convolutional Neural Networks to Medical Image Contour Detection
• Generative Convolution Layer for Image Generation
• MintNet: Building Invertible Neural Networks with Masked Convolutions
• Task-Agnostic Dynamics Priors for Deep Reinforcement Learning
• ReDMark: Framework for Residual Diffusion Watermarking on Deep Networks
• Optimising longitudinal and lateral calorimeter granularity for software compensation in hadronic showers using deep neural networks
• Activate or Not: Learning Customized Activation
• The Edge of Depth: Explicit Constraints between Segmentation and Depth
• Norm-preserving Orthogonal Permutation Linear Unit Activation Functions (OPLU)
• Autonomous Deep Learning: A Genetic DCNN Designer for Image Classification
• Convolutional Recurrent Neural Network Based Progressive Learning for Monaural Speech Enhancement
• Conditional Driving from Natural Language Instructions
• Multilayer Perceptron and Geometric Albedo Spectra for Quick Parameter Estimations of Exoplanets
• Learning High Dynamic Range from Outdoor Panoramas
• Deep Global-Relative Networks for End-to-End 6-DoF Visual Localization and Odometry
• NDT: Neual Decision Tree Towards Fully Functioned Neural Graph
• Design, Benchmarking and Explainability Analysis of a Game-Theoretic Framework towards Energy Efficiency in Smart Infrastructure
• Efficient Collaborative Multi-Agent Deep Reinforcement Learning for Large-Scale Fleet Management
• Revealing and Protecting Labels in Distributed Training
• Shifting Mean Activation Towards Zero with Bipolar Activation Functions
• Deep Learning applied to Road Traffic Speed forecasting
• Integrating Behavior Cloning and Reinforcement Learning for Improved Performance in Dense and Sparse Reward Environments
• Training Neural Networks by Using Power Linear Units (PoLUs)
• Breaking the Activation Function Bottleneck through Adaptive Parameterization
• Autoencoder-Based Incremental Class Learning without Retraining on Old Data
• Choice of training label matters: how to best use deep learning for quantitative MRI parameter estimation
• Causal-BALD: Deep Bayesian Active Learning of Outcomes to Infer Treatment-Effects from Observational Data
• Smooth activations and reproducibility in deep networks
• Adaptive First-and Zeroth-order Methods for Weakly Convex Stochastic Optimization Problems
• Timbre Analysis of Music Audio Signals with Convolutional Neural Networks
• flexgrid2vec: Learning Efficient Visual Representations Vectors
• On Loss Functions and Recurrency Training for GAN-based Speech Enhancement Systems
• The Impact of Global Structural Information in Graph Neural Networks Applications
• Classification Confidence Estimation with Test-Time Data-Augmentation
• Complexity Measures for Neural Networks with General Activation Functions Using Path-based Norms
• Jointly embedding the local and global relations of heterogeneous graph for rumor detection
• Multimodal Sensor Fusion In Single Thermal image Super-Resolution
• Regularizing by the Variance of the Activations' Sample-Variances
• Gated Hierarchical Attention for Image Captioning
• ALOHA: Auxiliary Loss Optimization for Hypothesis Augmentation
• Deep Learning Based Computed Tomography Whys and Wherefores
• LARNN: Linear Attention Recurrent Neural Network
• Versatile Auxiliary Regressor with Generative Adversarial network (VAR+GAN)
• Multiscale Hierarchical Convolutional Networks
• Topology-Aware Graph Pooling Networks
• Super-resolution Variational Auto-Encoders
• Enforcing exact physics in scientific machine learning: a data-driven exterior calculus on graphs
• CIFAR-10: KNN-based Ensemble of Classifiers
• Deep Residual 3D U-Net for Joint Segmentation and Texture Classification of Nodules in Lung
• Adaptive Weighting Multi-Field-of-View CNN for Semantic Segmentation in Pathology
• Transformation on Computer-Generated Facial Image to Avoid Detection by Spoofing Detector
• Chinese Typeface Transformation with Hierarchical Adversarial Network
• QR-MIX: Distributional Value Function Factorisation for Cooperative Multi-Agent Reinforcement Learning
• Image Correction via Deep Reciprocating HDR Transformation
• The Differentiable Cross-Entropy Method
• DuMLP-Pin: A Dual-MLP-dot-product Permutation-invariant Network for Set Feature Extraction
• Cross-Domain Perceptual Reward Functions
• Medical image super-resolution method based on dense blended attention network
• Additional Look into GAN-based Augmentation for Deep Learning COVID-19 Image Classification
• Multi-bin Trainable Linear Unit for Fast Image Restoration Networks
• An ELU Network with Total Variation for Image Denoising
• End-to-end Image Captioning Exploits Multimodal Distributional Similarity
• Dynamic Action Recognition: A convolutional neural network model for temporally organized joint location data
• Learning Dynamics from Noisy Measurements using Deep Learning with a Runge-Kutta Constraint
• Automatic Malware Description via Attribute Tagging and Similarity Embedding
• Approximation capabilities of neural networks on unbounded domains
• Imagined Value Gradients: Model-Based Policy Optimization with Transferable Latent Dynamics Models
• Deep Stacked Networks with Residual Polishing for Image Inpainting
• A Variational U-Net for Weather Forecasting
• An Attention-Gated Convolutional Neural Network for Sentence Classification
• Music Artist Classification with Convolutional Recurrent Neural Networks
• A Recursive Network with Dynamic Attention for Monaural Speech Enhancement
• Skeletal Movement to Color Map: A Novel Representation for 3D Action Recognition with Inception Residual Networks
• Classifying the classifier: dissecting the weight space of neural networks
• Tricks from Deep Learning
• Imitating Driver Behavior with Generative Adversarial Networks
• Video Frame Interpolation by Plug-and-Play Deep Locally Linear Embedding
• Channel Equilibrium Networks for Learning Deep Representation
• DPRed: Making Typical Activation and Weight Values Matter In Deep Learning Computing
• Learning Manifold Patch-Based Representations of Man-Made Shapes
• Training without training data: Improving the generalizability of automated medical abbreviation disambiguation
• CNN depth analysis with different channel inputs for Acoustic Scene Classification
• Towards End-to-End Audio-Sheet-Music Retrieval
• Improving Sequential Latent Variable Models with Autoregressive Flows
• Two Local Models for Neural Constituent Parsing
• Semi-Supervised Empirical Risk Minimization: Using unlabeled data to improve prediction
• A Deep Learning and Gamification Approach to Energy Conservation at Nanyang Technological University
• A General Framework Combining Generative Adversarial Networks and Mixture Density Networks for Inverse Modeling in Microstructural Materials Design
• The Adversarial Attack and Detection under the Fisher Information Metric
• Light-weight pixel context encoders for image inpainting
• Multi-channel Replay Speech Detection using an Adaptive Learnable Beamformer
• Automatic Music Highlight Extraction using Convolutional Recurrent Attention Networks
• Multi-Channel Auto-Encoder for Speech Emotion Recognition
• Effectiveness of Scaled Exponentially-Regularized Linear Units (SERLUs)
• Assessing Intelligence in Artificial Neural Networks
• NLS: an accurate and yet easy-to-interpret regression method
• Session-based Sequential Skip Prediction via Recurrent Neural Networks
• Learning Compact Neural Networks Using Ordinary Differential Equations as Activation Functions
• Direction is what you need: Improving Word Embedding Compression in Large Language Models
• AI based Algorithms of Path Planning, Navigation and Control for Mobile Ground Robots and UAVs
• MaCow: Masked Convolutional Generative Flow
• Rethink ReLU to Training Better CNNs
• Automated Polysomnography Analysis for Detection of Non-Apneic and Non-Hypopneic Arousals using Feature Engineering and a Bidirectional LSTM Network
• Avoiding Resentment Via Monotonic Fairness
• Modeling Sequences as Distributions with Uncertainty for Sequential Recommendation
• LQResNet: A Deep Neural Network Architecture for Learning Dynamic Processes
• CB2CF: A Neural Multiview Content-to-Collaborative Filtering Model for Completely Cold Item Recommendations
• Self-Supervised Generative Adversarial Network for Depth Estimation in Laparoscopic Images
• Gleason Score Prediction using Deep Learning in Tissue Microarray Image
• Beyond Point Estimate: Inferring Ensemble Prediction Variation from Neuron Activation Strength in Recommender Systems
• Integrated Decision and Control: Towards Interpretable and Computationally Efficient Driving Intelligence
• TanhSoft -- a family of activation functions combining Tanh and Softplus
• Modeling Document Interactions for Learning to Rank with Regularized Self-Attention
• Predicting traffic overflows on private peering
• Gaussian Process Neurons Learn Stochastic Activation Functions
• DelugeNets: Deep Networks with Efficient and Flexible Cross-layer Information Inflows
• Factorized Bilinear Models for Image Recognition
• Symmetrical Gaussian Error Linear Units (SGELUs)
• Natural-Logarithm-Rectified Activation Function in Convolutional Neural Networks
• Towards Label-Free 3D Segmentation of Optical Coherence Tomography Images of the Optic Nerve Head Using Deep Learning
• Quad-networks: unsupervised learning to rank for interest point detection
• TRk-CNN: Transferable Ranking-CNN for image classification of glaucoma, glaucoma suspect, and normal eyes
• ARiA: Utilizing Richard's Curve for Controlling the Non-monotonicity of the Activation Function in Deep Neural Nets
• DF-Net: Unsupervised Joint Learning of Depth and Flow using Cross-Task Consistency
• On the Universality of the Double Descent Peak in Ridgeless Regression
• Neural Multi-Step Reasoning for Question Answering on Semi-Structured Tables
• Dynamic Attention Based Generative Adversarial Network with Phase Post-Processing for Speech Enhancement
• Shapley Interpretation and Activation in Neural Networks
• A Sensitivity Analysis of Attention-Gated Convolutional Neural Networks for Sentence Classification
• Decomposition of Nonlinear Dynamical Systems Using Koopman Gramians
• Neural Random Projections for Language Modelling
• Group Anomaly Detection using Deep Generative Models
• The Effects of Noisy Labels on Deep Convolutional Neural Networks for Music Tagging
• Towards Deep Physical Reservoir Computing Through Automatic Task Decomposition And Mapping
• Generating Nontrivial Melodies for Music as a Service
• Stage-based Hyper-parameter Optimization for Deep Learning
• Nonconvex sparse regularization for deep neural networks and its optimality
• Reconstruction of Simulation-Based Physical Field by Reconstruction Neural Network Method
• Dueling Decoders: Regularizing Variational Autoencoder Latent Spaces
• Clinical Predictive Models for COVID-19: Systematic Study
• LIDAR-Camera Fusion for Road Detection Using Fully Convolutional Neural Networks
• Piecewise Linear Units Improve Deep Neural Networks
• Versatile Auxiliary Classifier with Generative Adversarial Network (VAC+GAN)
• Learning Less-Overlapping Representations
• Health-Informed Policy Gradients for Multi-Agent Reinforcement Learning
• ReGO: Reference-Guided Outpainting for Scenery Image
• Leveraging Class Similarity to Improve Deep Neural Network Robustness
• A Modified Activation Function with Improved Run-Times For Neural Networks
• OrthoSeg: A Deep Multimodal Convolutional Neural Network for Semantic Segmentation of Orthoimagery
• Sisyphus: A Cautionary Tale of Using Low-Degree Polynomial Activations in Privacy-Preserving Deep Learning
• Joint Parameter Discovery and Generative Modeling of Dynamic Systems
• Autonomous Navigation in Dynamic Environments: Deep Learning-Based Approach
• Decoder Choice Network for Meta-Learning
• Score-informed syllable segmentation for a cappella singing voice with convolutional neural networks
• Dual-Attention Graph Convolutional Network
• Investigating Transfer Learning Capabilities of Vision Transformers and CNNs by Fine-Tuning a Single Trainable Block
• ANS: Adaptive Network Scaling for Deep Rectifier Reinforcement Learning Models
• A Survey of Techniques All Classifiers Can Learn from Deep Networks: Models, Optimizations, and Regularization
• Modeling Information Flow Through Deep Neural Networks
• Applying Data Augmentation to Handwritten Arabic Numeral Recognition Using Deep Learning Neural Networks
• Teaching deep neural networks to localize single molecules for super-resolution microscopy
• Arbitrary Marginal Neural Ratio Estimation for Simulation-based Inference
• Capsule Network Performance with Autonomous Navigation
• Elementary superexpressive activations
• Expertise and confidence explain how social influence evolves along intellective tasks
• Semantic Segmentation of Earth Observation Data Using Multimodal and Multi-scale Deep Networks
• Decoupled Spatial Temporal Graphs for Generic Visual Grounding
• GeoT: A Geometry-aware Transformer for Reliable Molecular Property Prediction and Chemically Interpretable Representation Learning
• Acceleration of Radiation Transport Solves Using Artificial Neural Networks
• CRAM: Clued Recurrent Attention Model
• Prostate Segmentation using 2D Bridged U-net
• Real-time simulation of parameter-dependent fluid flows through deep learning-based reduced order models
• Structure-Based Networks for Drug Validation
• Deep-learning inversion: a next generation seismic velocity-model building method
• BORE: Bayesian Optimization by Density-Ratio Estimation
• Real-time Data Driven Precision Estimator for RAVEN-II Surgical Robot End Effector Position
• Reborn Mechanism: Rethinking the Negative Phase Information Flow in Convolutional Neural Network
• From Image to Imuge: Immunized Image Generation
• Learning structure-from-motion from motion
• Entity Candidate Network for Whole-Aware Named Entity Recognition
• Automatic Differentiation Variational Inference with Mixtures
• Discrete flow posteriors for variational inference in discrete dynamical systems
• Dynamic Weight Alignment for Temporal Convolutional Neural Networks
• Improving Fair Predictions Using Variational Inference In Causal Models
• Visual Motion Imagery Classification with Deep Neural Network based on Functional Connectivity
• Fractional moment-preserving initialization schemes for training deep neural networks
• Deep Global-Connected Net With The Generalized Multi-Piecewise ReLU Activation in Deep Learning
• Hippo: Taming Hyper-parameter Optimization of Deep Learning with Stage Trees
• High-resolution, yet statistically relevant, analysis of damage in DP steel using artificial intelligence
• Approximating Poker Probabilities with Deep Learning
• Reconstructing NBA Players
• Neural Material: Learning Elastic Constitutive Material and Damping Models from Sparse Data
• Alignment Attention by Matching Key and Query Distributions
• Learning OFDM Waveforms with PAPR and ACLR Constraints
• Convolutional Neural Networks: A Binocular Vision Perspective
• Learn-able parameter guided Activation Functions
• Quasi-Autoregressive Residual (QuAR) Flows
• Using Non-Linear Causal Models to Study Aerosol-Cloud Interactions in the Southeast Pacific
• Signature-Graph Networks
• GCCN: Global Context Convolutional Network
• Mining Insights on Metal-Organic Framework Synthesis from Scientific Literature Texts
• Orthogonal-Padé Activation Functions: Trainable Activation functions for smooth and faster convergence in deep networks
• Performance of artificial neural networks in an inverse problem of laser beam diagnostics
• A Convergence Theory Towards Practical Over-parameterized Deep Neural Networks
• Learning Low-Dimensional Quadratic-Embeddings of High-Fidelity Nonlinear Dynamics using Deep Learning
• A New Multifractal-based Deep Learning Model for Text Mining
• NeuroBack: Improving CDCL SAT Solving using Graph Neural Networks
• Learning invariance preserving moment closure model for Boltzmann-BGK equation
• Learning to Predict Diverse Human Motions from a Single Image via Mixture Density Networks
• HyperHyperNetworks for the Design of Antenna Arrays
• Energy-Based Anomaly Detection and Localization
• Social Influence Prediction with Train and Test Time Augmentation for Graph Neural Networks
• Causal Mediation Analysis with Hidden Confounders
• Single-Agent Optimization Through Policy Iteration Using Monte-Carlo Tree Search
• Deep Multi-task Network for Delay Estimation and Echo Cancellation
• Bayesian Attention Modules
• A Generative Model based Adversarial Security of Deep Learning and Linear Classifier Models
• Overcoming Overfitting and Large Weight Update Problem in Linear Rectifiers: Thresholded Exponential Rectified Linear Units
• Representation Learning for Sequence Data with Deep Autoencoding Predictive Components
• Permutation Matters: Anisotropic Convolutional Layer for Learning on Point Clouds
• Exploiting Review Neighbors for Contextualized Helpfulness Prediction
• Efficient Proximal Mapping of the 1-path-norm of Shallow Networks
• Adma: A Flexible Loss Function for Neural Networks
• Dereverberation using joint estimation of dry speech signal and acoustic system
• Classification of Visual Perception and Imagery based EEG Signals Using Convolutional Neural Networks
• Decoupling Global and Local Representations via Invertible Generative Flows
• Regularized Flexible Activation Function Combinations for Deep Neural Networks
• Segmenting Ships in Satellite Imagery With Squeeze and Excitation U-Net
• CALC2.0: Combining Appearance, Semantic and Geometric Information for Robust and Efficient Visual Loop Closure
• SentiMATE: Learning to play Chess through Natural Language Processing
• Towards Universal End-to-End Affect Recognition from Multilingual Speech by ConvNets
• Generalized Batch Normalization: Towards Accelerating Deep Neural Networks
• A Domain Generalization Perspective on Listwise Context Modeling
• Integrating Multiple Receptive Fields through Grouped Active Convolution
• Learning Socially Appropriate Robot Approaching Behavior Toward Groups using Deep Reinforcement Learning
• Covfefe: A Computer Vision Approach For Estimating Force Exertion
• An Exploration of Mimic Architectures for Residual Network Based Spectral Mapping
• VERAM: View-Enhanced Recurrent Attention Model for 3D Shape Classification
• Adversarial Decomposition of Text Representation
• A New Benchmark and Progress Toward Improved Weakly Supervised Learning
• Using General Adversarial Networks for Marketing: A Case Study of Airbnb
• Deep learning improved by biological activation functions
• Object Counts! Bringing Explicit Detections Back into Image Captioning
• Optimizing Sponsored Search Ranking Strategy by Deep Reinforcement Learning
• A Fully Trainable Network with RNN-based Pooling
• Audio to score matching by combining phonetic and duration information
• Orthogonal and Idempotent Transformations for Learning Deep Neural Networks
• Single Image Super-Resolution Using Lightweight CNN with Maxout Units
• InterpoNet, A brain inspired neural network for optical flow dense interpolation
• Learning Discrete Energy-based Models via Auxiliary-variable Local Exploration
• Robust Dual View Deep Agent
• Distilling with Residual Network for Single Image Super Resolution
• Towards Arbitrary Noise Augmentation - Deep Learning for Sampling from Arbitrary Probability Distributions
• High Diversity Attribute Guided Face Generation with GANs
• Monocular Depth Estimation with Directional Consistency by Deep Networks
• Predicting Auditory Spatial Attention from EEG using Single- and Multi-task Convolutional Neural Networks
• An Empirical Evaluation Study on the Training of SDC Features for Dense Pixel Matching
• Segmentation of Microscopy Data for finding Nuclei in Divergent Images
• SDC - Stacked Dilated Convolution: A Unified Descriptor Network for Dense Matching Tasks
• IC Neuron: An Efficient Unit to Construct Neural Networks
• Sequenced-Replacement Sampling for Deep Learning
• Unsupervised Rank-Preserving Hashing for Large-Scale Image Retrieval
• Attentive Long Short-Term Preference Modeling for Personalized Product Search
• The SWAG Algorithm; a Mathematical Approach that Outperforms Traditional Deep Learning. Theory and Implementation
• 3D Scene Parsing via Class-Wise Adaptation
• A Constructive Approach for One-Shot Training of Neural Networks Using Hypercube-Based Topological Coverings
• FSD: Feature Skyscraper Detector for Stem End and Blossom End of Navel Orange
• Motor Imagery Classification of Single-Arm Tasks Using Convolutional Neural Network based on Feature Refining
• An EEG-based Image Annotation System
• Convolutional Quantum-Like Language Model with Mutual-Attention for Product Rating Prediction
• Single Image Super-resolution via Dense Blended Attention Generative Adversarial Network for Clinical Diagnosis
• Deep Unsupervised Drum Transcription
• Deep Contextualized Self-training for Low Resource Dependency Parsing
• Random Bias Initialization Improves Quantized Training
• Nearly Minimal Over-Parametrization of Shallow Neural Networks
• The Quo Vadis submission at Traffic4cast 2019
• Farkas layers: don't shift the data, fix the geometry
• Photon-Driven Neural Path Guiding
• Goldilocks Neural Networks
• Modular Continual Learning in a Unified Visual Environment
• Deep Neural Networks with Short Circuits for Improved Gradient Learning
• Residual-Guided Learning Representation for Self-Supervised Monocular Depth Estimation
• RocNet: Recursive Octree Network for Efficient 3D Deep Representation
• Graph Representation Learning Network via Adaptive Sampling
• Explainable Recommendations via Attentive Multi-Persona Collaborative Filtering
• Permutation invariant networks to learn Wasserstein metrics
• EIS -- a family of activation functions combining Exponential, ISRU, and Softplus
• Continuous Representation of Molecules Using Graph Variational Autoencoder
• Generation and Simulation of Yeast Microscopy Imagery with Deep Learning
• Research on Optimization Method of Multi-scale Fish Target Fast Detection Network
• Input Invex Neural Network
• Pyramidal RoR for Image Classification
• DEBOSH: Deep Bayesian Shape Optimization
• Parametric Variational Linear Units (PVLUs) in Deep Convolutional Networks
• Solving time-dependent parametric PDEs by multiclass classification-based reduced order model
• Compression Method for Solar Polarization Spectra Collected from Hinode SOT/SP Observations
• Interpretable Option Discovery using Deep Q-Learning and Variational Autoencoders
• Hiding Images into Images with Real-world Robustness
• CleftNet: Augmented Deep Learning for Synaptic Cleft Detection from Brain Electron Microscopy
• Single Image Depth Prediction with Wavelet Decomposition
• Incorporating NODE with Pre-trained Neural Differential Operator for Learning Dynamics
• Embracing Ambiguity: Shifting the Training Target of NLI Models
• Intrusive deconvolutional neural networks for enhancing PIC/FLIP solutions
• Generating Data Augmentation samples for Semantic Segmentation of Salt Bodies in a Synthetic Seismic Image Dataset
• Group-Structured Adversarial Training
• Prediction of Hereditary Cancers Using Neural Networks
• Speech Imagery Classification using Length-Wise Training based on Deep Learning
• Applying supervised and reinforcement learning methods to create neural-network-based agents for playing StarCraft II
• Segmentation of Roads in Satellite Images using specially modified U-Net CNNs
• TaylorGAN: Neighbor-Augmented Policy Update for Sample-Efficient Natural Language Generation
• An Effective Training Method For Deep Convolutional Neural Network
• A Useful Motif for Flexible Task Learning in an Embodied Two-Dimensional Visual Environment
• Unsupervised learning of the brain connectivity dynamic using residual D-net