Improving Deep Neural Networks with Probabilistic Maxout Units

Abstract

We present a probabilistic variant of the recently introduced maxout unit. The success of deep neural networks utilizing maxout can partly be attributed to favorable performance under dropout, when compared to rectified linear units. It however also depends on the fact that each maxout unit performs a pooling operation over a group of linear transformations and is thus partially invariant to changes in its input. Starting from this observation we ask the question: Can the desirable properties of maxout units be preserved while improving their invariance properties ? We argue that our probabilistic maxout (probout) units successfully achieve this balance. We quantitatively verify this claim and report classification performance matching or exceeding the current state of the art on three challenging image classification benchmarks (CIFAR-10, CIFAR-100 and SVHN).

References in corpus (2)

• Practical Bayesian Optimization of Machine Learning Algorithms
• Stochastic Pooling for Regularization of Deep Convolutional Neural Networks

Cited by in corpus (18)

• Striving for Simplicity: The All Convolutional Net
• A survey on modern trainable activation functions
• Deep convolutional neural networks for brain image analysis on magnetic resonance imaging: a review
• Towards Dropout Training for Convolutional Neural Networks
• Recent Advances in Convolutional Neural Networks
• ReNet: A Recurrent Neural Network Based Alternative to Convolutional Networks
• Generalizing Pooling Functions in Convolutional Neural Networks: Mixed, Gated, and Tree
• Review: Deep Learning in Electron Microscopy
• APAC: Augmented PAttern Classification with Neural Networks
• Deep Learning with S-shaped Rectified Linear Activation Units
• Training Skinny Deep Neural Networks with Iterative Hard Thresholding Methods
• HD-CNN: Hierarchical Deep Convolutional Neural Network for Large Scale Visual Recognition
• CIFAR-10: KNN-based Ensemble of Classifiers
• Faster Convergence in Deep-Predictive-Coding Networks to Learn Deeper Representations
• Collaborative Layer-wise Discriminative Learning in Deep Neural Networks
• Unsupervised Feature Learning with C-SVDDNet
• Deep Global-Connected Net With The Generalized Multi-Piecewise ReLU Activation in Deep Learning
• Convolution in Convolution for Network in Network