Stochastic Pooling for Regularization of Deep Convolutional Neural Networks

Abstract

We introduce a simple and effective method for regularizing large convolutional neural networks. We replace the conventional deterministic pooling operations with a stochastic procedure, randomly picking the activation within each pooling region according to a multinomial distribution, given by the activities within the pooling region. The approach is hyper-parameter free and can be combined with other regularization approaches, such as dropout and data augmentation. We achieve state-of-the-art performance on four image datasets, relative to other approaches that do not utilize data augmentation.

9 pages

References in corpus (1)

• Improving neural networks by preventing co-adaptation of feature detectors

Cited by in corpus (141)

• Striving for Simplicity: The All Convolutional Net
• Do Deep Nets Really Need to be Deep?
• Deeply-Supervised Nets
• An Introduction to Convolutional Neural Networks
• Random Erasing Data Augmentation
• Compressing Neural Networks with the Hashing Trick
• Pre-Processing-Free Gear Fault Diagnosis Using Small Datasets with Deep Convolutional Neural Network-Based Transfer Learning
• Large-Margin Softmax Loss for Convolutional Neural Networks
• Towards Dropout Training for Convolutional Neural Networks
• Supervised Learning of Semantics-Preserving Hash via Deep Convolutional Neural Networks
• Recent Advances in Convolutional Neural Networks
• Systematic evaluation of CNN advances on the ImageNet
• Training deep neural networks with low precision multiplications
• Deep Convolutional Ranking for Multilabel Image Annotation
• ReNet: A Recurrent Neural Network Based Alternative to Convolutional Networks
• GridMask Data Augmentation
• ECG arrhythmia classification using a 2-D convolutional neural network
• Pooling Methods in Deep Neural Networks, a Review
• Regularization for Deep Learning: A Taxonomy
• Understanding and Improving Convolutional Neural Networks via Concatenated Rectified Linear Units
• Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networks
• A Comprehensive Overview and Comparative Analysis on Deep Learning Models: CNN, RNN, LSTM, GRU
• Data Augmentation in Natural Language Processing: A Novel Text Generation Approach for Long and Short Text Classifiers
• Deep Networks with Internal Selective Attention through Feedback Connections
• Swapout: Learning an ensemble of deep architectures
• Deep Learning of Representations: Looking Forward
• Generalizing Pooling Functions in Convolutional Neural Networks: Mixed, Gated, and Tree
• B-CNN: Branch Convolutional Neural Network for Hierarchical Classification
• APAC: Augmented PAttern Classification with Neural Networks
• What Do We Understand About Convolutional Networks?
• Application of Deep Learning methods to analysis of Imaging Atmospheric Cherenkov Telescopes data
• Dynamic Routing Between Capsules
• Improving Deep Neural Networks with Probabilistic Maxout Units
• Batch-normalized Maxout Network in Network
• Fast, simple and accurate handwritten digit classification by training shallow neural network classifiers with the 'extreme learning machine' algorithm
• Robust Convolutional Neural Networks under Adversarial Noise
• Bridging the Gap between Stochastic Gradient MCMC and Stochastic Optimization
• An empirical analysis of dropout in piecewise linear networks
• Training Skinny Deep Neural Networks with Iterative Hard Thresholding Methods
• Lets keep it simple, Using simple architectures to outperform deeper and more complex architectures
• Automated detection of vulnerable plaque in intravascular ultrasound images
• Multi-Bias Non-linear Activation in Deep Neural Networks
• Deep Learning Convolutional Networks for Multiphoton Microscopy Vasculature Segmentation
• Probabilistic Binary Neural Networks
• HD-CNN: Hierarchical Deep Convolutional Neural Network for Large Scale Visual Recognition
• Efficiently utilizing complex-valued PolSAR image data via a multi-task deep learning framework
• Untangling Local and Global Deformations in Deep Convolutional Networks for Image Classification and Sliding Window Detection
• Competitive Multi-scale Convolution
• PatchNet: Hierarchical Deep Learning-Based Stable Patch Identification for the Linux Kernel
• Conditional Generative Moment-Matching Networks
• Fuzzy Pooling
• Noisy Softmax: Improving the Generalization Ability of DCNN via Postponing the Early Softmax Saturation
• Resolution learning in deep convolutional networks using scale-space theory
• Stacked Pooling: Improving Crowd Counting by Boosting Scale Invariance
• Max-margin Deep Generative Models
• Learning Kernel for Conditional Moment-Matching Discrepancy-based Image Classification
• Max-Pooling Dropout for Regularization of Convolutional Neural Networks
• Multi-View Deep Learning for Consistent Semantic Mapping with RGB-D Cameras
• Hierarchical Deep Learning Architecture For 10K Objects Classification
• Object Recognition Using Deep Neural Networks: A Survey
• Learning ordered pooling weights in image classification
• Adversarial Examples Detection in Deep Networks with Convolutional Filter Statistics
• A Deep Generative Deconvolutional Image Model
• Manifold Regularized Deep Neural Networks using Adversarial Examples
• Very Deep Convolutional Neural Networks for Robust Speech Recognition
• A Review of Meta-Reinforcement Learning for Deep Neural Networks Architecture Search
• Wavelet Integrated CNNs for Noise-Robust Image Classification
• Convolutional Residual Memory Networks
• Numerical Methods and Analysis via Random Field Based Malliavin Calculus for Backward Stochastic PDEs
• LiftPool: Bidirectional ConvNet Pooling
• Multipartite Pooling for Deep Convolutional Neural Networks
• A fusion method for multi-valued data
• Automatic Configuration of Deep Neural Networks with EGO
• Training Neural Networks by Using Power Linear Units (PoLUs)
• Deep Epitomic Convolutional Neural Networks
• Ordinal Pooling Networks: For Preserving Information over Shrinking Feature Maps
• CIFAR-10: KNN-based Ensemble of Classifiers
• From Maxout to Channel-Out: Encoding Information on Sparse Pathways
• Multi Layer Neural Networks as Replacement for Pooling Operations
• Towards accurate quantitative photoacoustic imaging: learning vascular blood oxygen saturation in 3D
• Latent Model Ensemble with Auto-localization
• Parallel Grid Pooling for Data Augmentation
• M${}^2$S-Net: Multi-Modal Similarity Metric Learning based Deep Convolutional Network for Answer Selection
• Drop-Activation: Implicit Parameter Reduction and Harmonic Regularization
• On the Importance of Normalisation Layers in Deep Learning with Piecewise Linear Activation Units
• Invariant backpropagation: how to train a transformation-invariant neural network
• No more meta-parameter tuning in unsupervised sparse feature learning
• End-to-End Learning Local Multi-view Descriptors for 3D Point Clouds
• A Generative Model for Deep Convolutional Learning
• Long Short-Term Memory based Convolutional Recurrent Neural Networks for Large Vocabulary Speech Recognition
• Learning Discriminative Features via Label Consistent Neural Network
• Collaborative Layer-wise Discriminative Learning in Deep Neural Networks
• Differentially Private Bayesian Neural Networks on Accuracy, Privacy and Reliability
• Inference with Hybrid Bio-hardware Neural Networks
• Generative Deep Deconvolutional Learning
• CUNet: A Compact Unsupervised Network for Image Classification
• Orthant Based Proximal Stochastic Gradient Method for ${\ell }_1$-Regularized Optimization
• Adaptive Normalized Risk-Averting Training For Deep Neural Networks
• Simplified Stochastic Feedforward Neural Networks
• Universal Pooling -- A New Pooling Method for Convolutional Neural Networks
• Learning Machines Implemented on Non-Deterministic Hardware
• Improving the Resolution of CNN Feature Maps Efficiently with Multisampling
• Self-Orthogonality Module: A Network Architecture Plug-in for Learning Orthogonal Filters
• Universum Prescription: Regularization using Unlabeled Data
• Improving training of deep neural networks via Singular Value Bounding
• Compression of Deep Neural Networks on the Fly
• Unsupervised Feature Learning with C-SVDDNet
• Understanding Dropout: Training Multi-Layer Perceptrons with Auxiliary Independent Stochastic Neurons
• Reducing Artificial Neural Network Complexity: A Case Study on Exoplanet Detection
• Convolutional Neural Network on Three Orthogonal Planes for Dynamic Texture Classification
• Recent Advances in Convolutional Neural Network Acceleration
• Learn Faster and Forget Slower via Fast and Stable Task Adaptation
• Deep Global-Connected Net With The Generalized Multi-Piecewise ReLU Activation in Deep Learning
• S3Pool: Pooling with Stochastic Spatial Sampling
• Max-Margin Deep Generative Models for (Semi-)Supervised Learning
• Adma: A Flexible Loss Function for Neural Networks
• A Fully Trainable Network with RNN-based Pooling
• Adaptive Regularization via Residual Smoothing in Deep Learning Optimization
• Deep Learning Estimation of Absorbed Dose for Nuclear Medicine Diagnostics
• Deep Collaborative Learning for Visual Recognition
• Alpha-Integration Pooling for Convolutional Neural Networks
• Learning Non-Parametric Invariances from Data with Permanent Random Connectomes
• Convolution in Convolution for Network in Network
• CNNs are Globally Optimal Given Multi-Layer Support
• Pseudo-positive regularization for deep person re-identification
• Nonparametric Online Learning Using Lipschitz Regularized Deep Neural Networks
• Input Fast-Forwarding for Better Deep Learning
• Geometric Neural Phrase Pooling: Modeling the Spatial Co-occurrence of Neurons
• Stochastic Region Pooling: Make Attention More Expressive
• Multiple Instance Learning Convolutional Neural Networks for Object Recognition
• Feature Fusion Detector for Semantic Cognition of Remote Sensing
• Efficient Modelling Across Time of Human Actions and Interactions
• Attention Incorporate Network: A network can adapt various data size
• Adopting Robustness and Optimality in Fitting and Learning
• Deep Trans-layer Unsupervised Networks for Representation Learning
• Multi-scale Convolution Aggregation and Stochastic Feature Reuse for DenseNets
• Class Interference Regularization
• Emergence of Selective Invariance in Hierarchical Feed Forward Networks
• Diagnosis of Pediatric Obstructive Sleep Apnea via Face Classification with Persistent Homology and Convolutional Neural Networks
• ECG beats classification via online sparse dictionary and time pyramid matching
• Regularizing Neural Networks via Stochastic Branch Layers